Beyond the QA Bottleneck

AI adoption brought verification into the development loop

I made AI-assisted development include test authoring, device verification and evidence for review. From May through September 2026, the app and its design system recorded 921 merged PRs across both repositories, alongside a sustained build-out of the test suite. September alone recorded 188 mobile PRs. The earlier process had tests, but verification arrived too late: unit-test files moved from 93 in January to 113 in April, while Maestro stayed at 14 YAML files. Those device flows had existed since July 2024, yet they were not running in CI.

Finishing implementation left a familiar question: what else needed checking before we could release it? A nearby change could bring an already tested screen back into scope. Regressions weakened confidence, and manual QA carried repeated checks that did not leave an executable record for the next release. Useful backlog improvements became harder to schedule because the time between writing the code and trusting the result was difficult to predict.

I joined in April 2026 and started changing that process in May. The earlier chart history gives context for what followed. By 8 October, the app had 2,257 unit-test files and 404 Maestro YAML files, with smoke runs, device captures, review automation and installable test builds around the work.

My goal was to make AI adoption useful all the way through development. I wanted the agent to understand the requirement, implement it, write meaningful tests, run the checks and show the interaction on a device. That would give me something concrete to review. The change to the process mattered as much as the ability to generate code: each task needed to leave behind evidence and reusable protection.

We also moved towards product engineering. A smaller engineering team took on broader responsibility for product decisions, mobile development and the web experience. In practice, I saw features become faster to develop and deliver, with more work completed across that wider scope. That is my experience of the transition; the activity charts below provide a visible record alongside it.

The rhythm of development changed from May

The monthly record follows app development from July 2024 through 8 October 2026. October is partial and excluded from the complete-month averages below. From May 2026, the series combines the original app repository with PRs touching the app or its design system in the later monorepo. Unrelated web and infrastructure work is excluded. January through September therefore gives a view of mobile development around the May change in process. Its merged-PR counts were 25, 32, 46, 58 from January to April, followed by 159, 222, 158, 194, 188 from May to September. The app moved repositories in September; that full month combines 77 original-repository PRs with 111 app-scoped monorepo PRs. This also revises May to August upward from the narrower, original-repository-only series. For longer-term context, the 2025 monthly counts ranged from 21 to 92.

January through April spans 4 complete months and 161 merged PRs; May through September spans 5 complete months and 921 PRs. To account for the unequal window lengths, I compare monthly averages: 40.25 and 184.20 respectively, about 4.58 times as many merges per month. This records a sustained increase in merge activity. Changes differ in size and purpose, so the ratio cannot be read as the same increase in effort, productivity, features delivered or customer value.

I also count distinct ticket identifiers in each month. They moved from 22, 27, 33, 50 in January to April to 137, 205, 109, 143, 154 in May to September. These windows contain four and five months respectively. The sums are 132 and 748 monthly ticket occurrences. A ticket spanning two months can be counted in both, so I do not present those sums as unique features. This second measure helps check whether higher PR activity was accompanied by work against more tracked requests.

The October checkpoint records 126 merged mobile PRs and 77 distinct tickets through 8 October. Of those PRs, 79 were titled as test work, 22 as fixes and 17 as features. This makes the test investment visible; the partial month is not comparable with a full month. PR counts are the clearest continuous record of development activity I can share here. I use them to show the rhythm of work moving through review, not as a productivity target. Ten PRs can involve more effort than fifteen, or less. Planning, investigation, implementation, testing and product decisions also happen without a merge, so a day with none can still contain substantial work. Splitting a feature into smaller PRs changes the count too. These monthly charts show recorded merge activity; they do not measure daily effort, feature completion or turnaround.

The outcome I cared about was how quickly we could develop and deliver useful features with confidence. I experienced an improvement there as the loop became established, even as the team became smaller and its product and web responsibilities grew. The PR history documents visible activity accompanying that change; it does not quantify the time from a feature request to its release or establish an improvement per engineer.

This is an observational comparison: team size, responsibilities, task mix, scope, AI assistance and the engineering process changed together. It does not isolate AI's contribution. The useful question is how we made the work assessable: which checks ran, what evidence accompanied it, and what remained uncertain at review.

I gave implementation and review the same rulebook

The first app rulebook appeared on 7 May. On 24 and 25 July, I made the standards explicit, moved lint rules from warnings to errors, connected the AI reviewers to those standards and added a self-verification runbook. The agent now had a contract to follow across different tasks.

The rules covered small files with focused responsibilities, business logic in hooks and helpers, reuse of existing components and tests for changed production code. The target was roughly 100 lines per file, rather than allowing one large UI file to accumulate every decision. Existing exceptions remained, so I treat this as an enforced direction for new work rather than claiming that the entire application suddenly met every standard.

The agent writes a plan in the task description before implementation. I can correct a misunderstanding while it is still a paragraph, before it becomes a feature to unwind. Focused skills then carry the approved task through the remaining work.

The same rules reach the reviewers. CodeRabbit reads the main guidance file; Qodo reads a mirror plus explicit compliance checks; cubic uses focused review rules. A small CI comparison fails when the mirror drifts from the main rulebook. Implementation and review use the same version of the contract.

Eight focused skills carry the task through the loop

I packaged the process as eight named skills with specific responsibilities. A skill is a repeatable set of instructions that an agent loads for a job. Cook coordinates the workflow, and saul keeps communication clear throughout. The other skills perform focused stages, each leaving a concrete output for the next one.

The handoffs keep autonomy bounded. Research returns findings from the ticket, discussions, documents, design and code; planning turns them into a checklist. Once I approve it, implementation works in a fresh branch and worktree from the main branch. Parallel tasks keep their checkouts and devices separate.

Capture and review have their own instructions and evidence requirements. Human review still controls acceptance and merge. A later run confirms the merge before clearing the temporary worktree, branch and local evidence and releasing the devices. Worktree creation and cleanup are lifecycle stages around the eight named skills.

This split also gives new lessons a home. Flaky selectors update capture guidance; a build command updates discovery; a missed edge case updates implementation and review. The next task gets those instructions alongside the tests that the previous work left behind.

A task had to produce more than implementation

The workflow became plan, implement, test, verify, capture, review and deliver a test build. Its completion criteria extend beyond the code: a changed behaviour needs tests, a UI change needs device evidence, and a reviewer needs fresh results for the revision they are assessing.

The agent runs checks and reads their output before describing them as passing. Review material identifies what was exercised and its revision. If the branch changes, the affected checks need fresh results. Verification becomes part of finishing the task.

Device-evidence blocks started appearing in PRs on 19 June. Recent evidence tables link green device-flow runs to the revision under review. Those tables were still posted by the engineer, so I do not describe the whole evidence package as fully automatic.

This is the mechanism behind reducing repetitive QA. When a defect becomes a test, the next change can exercise that behaviour again. When a recording accompanies the implementation, a reviewer can inspect the result without reconstructing the whole task. When a test build is available, someone can try it on a real device. Each artifact answers a particular question and reduces how much confidence must be assembled again at release time.

The test inventory grew as tests became part of each change

The unit-test history shows the change in habit over several months. January to April ended at 93, 95, 98, 113 files. The following month-end snapshots were 164 in May, 250 in June, 395 in July and 1,040 in August. The September month-end snapshot contains 1,669 files. By 8 October it reached 2,257 files. Compared with April, that latest snapshot contains 2,144 additional test files, about 20.0 times the earlier inventory.

These are test files, not individual assertions or a coverage percentage. A file can contain many cases, and splitting a file changes the inventory without increasing protection. I include the timeline because it makes the sustained investment visible; it needs to be read alongside the behaviours the tests exercise and the results of actually running them.

I made tests part of the implementation contract. The agent uses Jest with jest-expo and a shared render helper that supplies providers and routing. The test setup has 35 recorded MSW GraphQL handlers shared with 106 Storybook stories, so tests and visible states use the same controlled responses. Business decisions that do not need rendering live in pure helpers. That gives the agent a practical way to test a failure case without booting the entire interface.

For a regression, I ask for a test that fails before the fix and passes after it. I also temporarily revert the correction to confirm that the specification fails for the original reason. In one review, Qodo caught two defects that earlier passes had missed; both new specifications failed when their fixes were reverted. That establishes protection more clearly than a count or a test that follows the implementation.

Technique reference: React Native Testing Library

CI checks the change and reports what was exercised

The PR check chain includes Biome, TypeScript, unit tests, coverage, Expo export and prebuild, plus checks for the device flows and release tooling. AI agents write implementation and tests; these CI commands remain deterministic. The agent can run the chain, interpret a failure and correct the code, but its explanation cannot establish that a build passed. I want the actual output from the actual revision.

Coverage needs an explicit scope. File counts describe an inventory, while a coverage report describes code exercised during a particular test run. Expectations for changed production code and inherited project-wide checks concern different scopes. A green global result cannot establish complete application coverage or prove that every changed line met the task's expectations. I review the changed behaviour and its relevant test results together.

The September main-branch run measured 72.02% line coverage, with 1,351 suites and 7,693 tests. On 2 October, measured line coverage was 75.04%. The 8 October run, merged from four Jest shards, reached 99.26% lines, 98.96% statements, 95.55% branches and 98.1% functions. The global CI thresholds rose during 7 and 8 October and were pinned to the measured baseline, so subsequent changes must preserve it.

The increase includes new tests and a more explicit measurement scope. Covered lines rose from 19,178 to 22,903, while eligible lines fell from 25,555 to 23,072 because test helpers, Storybook, development-only code and other excluded files no longer enter the denominator. The reported percentage is 99.26% of that configured scope; it does not mean every application behaviour has been verified. Coverage reporting began in May, so there is no measured pre-May percentage to plot as zero.

I also separated dispatch success from test success. GitHub Actions triggers the native work in EAS; a dispatch job can finish in seconds while the build and flows continue elsewhere. Fresh scheduled device builds tie the run to a known commit. Their results need to be visible where someone makes the merge or release decision.

Maestro became a running system rather than a folder of flows

The Maestro inventory stayed at 14 YAML files from January through April, grew to 129 in May, then reached 135 in June, 136 in July and 147 in August. It reached 225 at the end of September and 404 by 8 October. The checkpoints show investment in reusable device checks over time; 404 files include several roles rather than 404 independent journeys.

The first EAS device workflow landed in May. On 18 June, the flows moved into Maestro Cloud with PR smoke and nightly execution. That was the important change from the earlier dormant folder: a test could now run alongside development and return a result. Making that useful required deliberate suite entrypoints, controlled setup and jobs that could finish within the device service's limits.

Technique reference: Maestro: how device flows work

Leaf flows, shared setup and aggregates have different jobs

A separately recorded file-role sample contains 125 leaf flows, 53 shared sub-flows and 37 suite aggregates. This is an earlier structural sample, not a breakdown of the 404 files at the 8 October checkpoint. Leaves exercise a behaviour, shared sub-flows reuse setup or common steps, and aggregates provide entrypoints that call the relevant leaves. Those role counts describe the structure; they are not an exhaustive, mutually exclusive partition of every YAML file or a count of independent journeys.

The 8 October route report reaches all 99 non-exempt routes through leaf flows in the inventory. Only 28 routes, 28.3%, are reached by leaves that CI runs; 71 are covered only by quarantined flows, and 12 routes are exempt. Inventory reachability therefore does not establish successful automated execution. Selection follows the call graph. Each cloud job names an aggregate entrypoint; tags on individual files do not decide which CI runs include them. A new leaf therefore needs to be connected to the entrypoint that should exercise it. Shared setup reduces repeated maintenance, but changing it can affect several callers. The static analyser checks paths and reachability so that adding a plausible file does not silently leave it outside execution.

Device execution needed platform scope and a time budget

By 8 October, the scheduled configuration reached 33 cloud jobs: 23 on iOS and 10 on Android. The earlier configuration had 31 jobs. These are configured suite jobs, rather than 33 passing tests or a coverage percentage. PR labels select a smoke run or the broader device suite. Fresh scheduled builds tie execution to a known commit, and each platform's results describe the work it actually ran.

We changed the structure after a flaky suite prevented the suites behind it from producing evidence. On 5 August, nightly execution was split into one job per suite and platform. Two days later, a nine-leaf run hit Maestro Cloud's 15-minute limit. Some leaves needed roughly two minutes of setup each, so we split long suites into parts of three leaves or fewer. A flow that fits the execution budget can finish and leave a useful result; grouping everything into one long run made failures harder to isolate.

Predictable setup mattered as much as grouping. The flows use controlled initial state and launch arguments to select test conditions, with backend scenarios for known responses rather than a separate mock server. Accessibility labels and exact text matching make selectors easier to diagnose, while overlays and sticky elements can still intercept taps. I want a failed flow explained and repaired. Changing test data until a run happens to pass leaves the original uncertainty in place.

I added static checks for the tests themselves

Growing the device suite exposed another problem: a file could look plausible and still be unable to run, or never be reached by a scheduled aggregate. On 5 August, I added a static analyser for the Maestro YAML. It checks syntax, paths, application identifiers, selectors that cannot resolve and which flow files the CI entrypoints can reach. That catches problems before an expensive build and device run.

The analyser has nine rules, eight of them blocking, and 104 self-test cases. It runs in under a second in the normal CI lint job. Exceptions live in an allow-list with reasons rather than being silently ignored. The distinction between an inventory file and a reachable, executable test became part of the tooling, rather than something a reviewer had to remember every time.

We documented failure patterns for agents: exact text matching, footers intercepting taps, overlays and incorrect initial state. The next agent reads those learnings before trying the same approach. Debugging updates the test and its maintenance instructions.

Release automation has its own self-tests, with 167 checks in the PR chain. Tags decide versions, and over-the-air updates must reject native changes. Those rules need executable specifications because a regression in the delivery machinery can affect the result too.

The review includes screenshots and a recording from both platforms

Device evidence runs as a second pass after implementation and checks. The agent launches the app, reaches the affected screen and exercises the change where it is actually used. If a flag gates the interface, it confirms the flag is enabled before recording. An isolated component example helps with states, but the evidence also needs to show the surrounding screen and interaction.

The workflow asks for multiple screenshots and an MP4 on each platform. One reviewed change attached twelve files: three screenshots and three recordings on iOS, and the same on Android. The iOS capture uses simulator recording converted to MP4; Android uses screenrecord and pulls the recording with adb. Screenshots show visible states, while recordings show waiting, failure or recovery. Platform and revision details travel with the captures so that a reviewer can understand what they represent.

iOS and Android capture can proceed in parallel with one worker per platform, provided each claims a separate device and pins its commands to that device identifier. This prevents two concurrent tasks from tapping or recording the same simulator. Recordings are attached to the task, and the PR links to that evidence. It gives the reviewer a direct path from the requested behaviour to what happened on the device.

Simulator captures show the interaction in that environment; TestFlight adds feedback from physical devices. Reviewers can inspect the recorded result and decide what exploratory checking remains, instead of beginning acceptance by asking the author to recreate the changed flow.

Shared components gave agents verified building blocks

The mobile design-system work started in May. At the snapshot it contained 94 components and 801 stories, with 213 test files. The later 8 October checkpoint records 827 story exports and 133 published design-system versions. A separate design-diff loop had marked 29 components as verified after proving less than 1% pixel difference against the design. The other 65 were outside that recorded state; this does not mean they were untested or failed. The app's rulebook directs new reusable UI into the shared package, giving agents an existing catalogue of behaviour and states.

One story serves several consumers: web Storybook through react-native-web, an on-device Storybook app, Chromatic snapshots and Vitest browser tests. The native catalogue is delivered through TestFlight, so a component can be exercised away from the developer's machine. Chromatic blocks a design-system change until its visual result is reviewed. Code, interaction, appearance and native behaviour each get their own evidence.

The Button examples illustrate the depth this can carry. Its catalogue has 24 stories including hover, press and focus interactions, alongside unit and interaction tests, a design reference and an accessibility report. Across the design-system tests, queries by role appear 494 times. Roles, labels and states are part of the component contract, so both assistive technology and automated tests have meaningful information to work with.

A shared-component change needs verification across its consumers, with appearance changes made explicit by the visual gate.

AI review became another pass over a concrete change

From 25 July, three AI reviewers read the development rules alongside human reviewers. In the recorded sample of the last 30 merged PRs, they wrote 360 of 635 inline comments, about 57%. The remaining 275 came from other reviewers. This sample covers the broader repository, including work beyond the app. It measures review activity, not comment correctness, bugs prevented or the share of implementation AI wrote.

The implementation agent evaluates each comment: inspect the code, reproduce the concern, change when justified and rerun the affected checks. A reviewer can reveal a missing failure case or misunderstand a deliberate choice. Replies need evidence; automatically accepting suggestions is not a quality gate.

The PR brings the requirement, implementation, test output, device captures and remaining limitations together. Human review controls acceptance and merge, with a concrete decision to make and a visible place to record it. Test builds extend assessment beyond the author, and TestFlight gives testers access on physical devices.

I established dedicated test-build workflows on 22 June. Their purpose is to prepare installable builds for feedback beyond local captures, using distribution appropriate to each platform. A completed submission gives testers another way to assess the change on a real device. Publishing a public store release remains a separate decision with its own steps.

Technique reference: Expo: distributing a build with TestFlight

A more active release history needed clear delivery milestones

The updated release history records four store-release tags in July, three in August and six in September. September combines two in the original app repository and four in the monorepo. These counts show a more active release line, but tags do not establish confirmed public-store go-live dates on both platforms. A tag, a production workflow run, a TestFlight submission and a public release are different events. I keep those distinctions visible rather than using those tag counts to claim multiple public releases every week.

On 11 and 12 September, I added release tooling and runbooks to make delivery more repeatable. Generated release notes derive from reviewed PR titles, and self-tests check the version and delivery decisions against the workflow configuration. The release machinery needs executable specifications just as the application does.

I separated the delivery milestones: creating a build, submitting it for testers, preparing a store release and making it public. Automation can support these steps, but success has to be established at the step itself. A successful dispatch is not confirmation that the downstream build or submission completed. A release-history entry therefore needs context before it becomes evidence of public availability.

Native store builds and over-the-air updates need different compatibility checks. I added tests around the release rules, including rejecting native changes from an update intended for an existing runtime. Human review and public-release decisions remain part of the process. The improvement was a repeatable delivery path and a clearer account of what each recorded event established.

The Sentry history became part of the development loop

On 28 July, I made error cleanup systematic and connected Sentry issues to development tickets. A resurfacing problem could return to the backlog instead of being rediscovered in an unrelated chat. Release context and active feature flags accompany events, helping explain which implementation the user encountered. The work then becomes reproduce, classify, fix where appropriate and add a test that preserves the correction.

The daily chart shows all-environment error events from the retained history. Its 86 complete days run from 7 July through 30 September. The 6 July boundary was missing its first hour, so I exclude it. The series stops at September to keep the incomplete current month out of the comparison. A very low count from a partially elapsed current day would otherwise make the apparent improvement look much larger than the evidence supports.

The 30 complete days from 7 July to 5 August contain 364,417 events. The same-length window from 1 to 30 September contains 152,809, about 58% fewer recorded events. Fixes, reporting filters and sampling changes all contributed; the data cannot separate their effects. All environments are included, and there is no session or traffic denominator. These are recorded event counts, not a crash-free rate or a measure of bugs eliminated by AI.

The 8 October refresh attributes 66 new issue types to September releases and none to October releases through that checkpoint. These are release-level issue attributions, not the daily event series, and attribution can change as later events arrive. Recorded volume rose again during September before settling, something a two-point chart would hide. Distinct issues and repeated events tell different stories: one recurring condition can dominate the count. I use these signals to prioritise investigations. Reproducible defects return to the same test, device-evidence and review loop.

Technique reference: Sentry: release health session statistics

Performance needed its own evidence and controls

On 26 August, I added a performance harness that samples main-thread activity during repeated list flings on a Release simulator build. It performs ten 300-millisecond flings, so repeated runs have a defined workload. One finding was that Session Replay accounted for 48% to 73% of UI-update work in that measurement. That figure describes the sampled workload; it is not a claim that disabling Replay made the whole application that much faster.

I introduced performance changes behind feature flags so they could be measured before temporary controls were retired. The source records 20 performance flags retired, including 14 on 22 September. The useful habit was to profile a known workload, make a bounded change and rerun the measurement before simplifying the implementation.

I also added static checks for patterns that had already created avoidable work: timers, expensive blur and gradient effects, autoplay without pause handling, large shadows and unstable style objects. The linter blocks new occurrences of eight selected patterns while documenting older exceptions. This turns a profiling lesson into a check that the next agent encounters before it repeats the same approach.

A local profile and a scheduled performance trend establish different things. The local finding describes a known workload and environment. Using scheduled measurements as a trend requires completed, repeatable runs with results that can be compared. Performance belongs in QA only when its workload, environment and results are visible enough to interpret; configuring a workflow alone does not establish protection for every release.

A useful loop also reports what it did not verify

The 404 Maestro files recorded on 8 October do not mean every path runs on every platform. They include shared setup and suite entrypoints, while the configured jobs cover different platform scopes. A green result needs to say which behaviours it exercised. Inventory growth and a passing subset cannot establish complete platform coverage.

Review needs completed device and performance results, rather than confirmation that a job was triggered. I want the result linked to its revision, platform and verification scope, where a reviewer can inspect it. If that evidence is absent, the review should say what remains unverified instead of inferring success from the workflow configuration.

The next improvements follow those scopes: broaden platform verification, keep test setup predictable, make device and performance results easy to inspect, and verify delivery paths from the reviewed change through their downstream result. Affected-only CI matters too, because the move from roughly five to fifteen minutes makes repeated verification more expensive. Each improvement needs a defined scope and an observable result.

What I take from these months is a practical way to adopt AI: define the work, give the agent clear rules, make it write and run meaningful tests, capture the interaction and put the evidence in front of a reviewer. The measured result was a more active app development record and a much larger verification inventory. In practice, feature development and delivery became faster while a smaller team took on broader product and web responsibilities. The lasting benefit is that the next task starts with tests, shared components and production learning that the previous work left behind.

Resources for putting this into practice

These official references explain the tools and measurement concepts. The observations in the figures are my anonymised aggregate records, not results reported by these documentation sources.

• Maestro: how device flows work

  Accessibility-driven interactions and reusable UI flows.

• React Native Testing Library

  Tests that exercise components through user-facing behaviour.

• Expo: distributing a build with TestFlight

  Build submission, tester access and the public-release boundary.

• Sentry: release health session statistics

  Session and user counts, crash rates and crash-free rates.