You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
\Cref{fig:model-comparison} summarizes both overall and category-level performance across eight primitive models. In overall weighted average score, \texttt{claude-opus-4.6} leads (84.3\%), followed by \texttt{claude-sonnet-4.6} (77.9\%) and \texttt{glm-5} (74.2\%), while \texttt{deepseek-r1-distill-qwen-32b} scores lowest (26.1\%). The category breakdown shows non-uniform behavior: navigation is generally the strongest category for most models, whereas animation and async-state are more discriminative and better separate mid-tier models. For each model, detailed analysis is provided in appendix section A. The \cref{tab:tokens-by-model} summarizes token usage for each of the models.
517
+
\Cref{fig:model-comparison} summarizes both overall and category-level performance across the evaluated solver models. In overall weighted average score, \texttt{composer-2} leads (96.1\%), followed by \texttt{composer-2-fast} (94.9\%) and \texttt{gpt-5.4} (85.3\%), while \texttt{deepseek-r1-distill-qwen-32b} scores lowest (44.4\%). The category breakdown remains non-uniform: navigation and React Native API tasks are generally the strongest categories across the model set, whereas animation remains the most discriminative category and lists provide additional separation among mid-tier models. For each model, detailed analysis is provided in appendix section A. The \cref{tab:tokens-by-model} summarizes token usage for each of the models.
506
518
507
519
\begin{figure}[htbp]
508
520
\centering
@@ -518,19 +530,24 @@ \section{Results}
518
530
\toprule
519
531
\textbf{Rank} & \textbf{Model} & \textbf{Weighted average score} & \textbf{Runs} \\
\caption{Overall model comparison from packed result archives.}
@@ -573,11 +590,11 @@ \section{Conflict of Interest}
573
590
The authors are affiliated with Callstack. The experiments used commercially available infrastructure acquired and operated independently; no direct sponsorship, funding, or editorial influence was provided for this work.
574
591
575
592
\section{Conclusion}
576
-
We evaluated \textbf{13} solver models on the React Native eval suite (\textbf{43} evals in total), with \textbf{10 repeated runs per model} under the same pipeline and scoring rules. Across completed runs, the highest mean \texttt{weightedAverageScore} was \textbf{0.844} (\textbf{84.4\%}, model: \texttt{claude-opus-4.6}); the median across models was \textbf{0.722} (\textbf{72.2\%}). Category-level analysis (\cref{fig:model-comparison}) shows strongest performance on \textbf{navigation} and weakest performance on \textbf{animation}, with \textbf{async state} in between and strongly differentiating mid-tier models. Recurring failures were concentrated in behavior-heavy requirements, especially animation dynamics and async-state consistency/invalidation logic.
593
+
We evaluated \textbf{18} solver models on the React Native eval suite (\textbf{66} evals in total), with \textbf{10 repeated runs per model} under the same pipeline and scoring rules. Across completed runs, the highest mean \texttt{weightedAverageScore} was \textbf{0.961} (\textbf{96.1\%}, model: \texttt{composer-2}); the median across models was \textbf{0.761} (\textbf{76.1\%}). Category-level analysis (\cref{fig:model-comparison}) shows strongest performance on \textbf{navigation} and \textbf{react native apis}, while \textbf{animation} remains the weakest and most discriminative category; \textbf{lists} and \textbf{async state} provide additional separation among otherwise closely clustered models. Recurring failures were concentrated in behavior-heavy requirements, especially animation dynamics and async-state consistency/invalidation logic.
577
594
578
595
Two practical implications follow. First, React Native development difficulty is not uniform: models that are competitive on navigation and structural code generation can still be unreliable on behavior-heavy work, where correctness depends on timing, gesture/animation dynamics, and state invalidation semantics. For practitioners, this suggests applying the most scrutiny (and the most automated protection via tests and runtime validation) to animation and async-state code paths, even when a model performs strongly overall. Second, requirement-level evaluation yields actionable signals: per-requirement verdict traces support targeted prompt iteration, scaffolding improvements, and dataset evolution without collapsing all behavior into a single task-level pass/fail.
579
596
580
-
From an operational standpoint, token usage varied substantially across models (\cref{tab:tokens-by-model}), ranging from \textbf{90,425} mean tokens per run (\texttt{gpt-5.4}) to \textbf{821,708} (\texttt{grok-4}). This highlights a cost/latency frontier that is not directly implied by score alone, and reinforces that model selection for production assistance should consider both category-specific performance and consumption characteristics.
597
+
From an operational standpoint, token usage varied substantially across models (\cref{tab:tokens-by-model}). Among models with available token aggregates, mean usage ranged from \textbf{254,437} tokens per run (\texttt{deepseek-r1-distill-qwen-32b}) to \textbf{5,590,132} (\texttt{deepseek-v3.2}), with additional high-consumption behavior visible for \texttt{kimi-k2.5} at \textbf{2,500,716}. The token totals for \texttt{composer-2} and \texttt{composer-2-fast} could not be determined precisely from the captured runs, so efficiency comparisons for those two models should be interpreted with caution. Overall, this highlights a cost/latency frontier that is not directly implied by score alone, and reinforces that model selection for production assistance should consider both category-specific performance and consumption characteristics.
581
598
582
599
Overall, these results indicate that React Native Evals separates model capability in a reproducible and auditable way while preserving requirement-level interpretability (via explicit per-requirement weights and per-eval verdict artifacts). They also highlight a need to extend the current evaluation set beyond the existing categories to more of the day-to-day React Native surface area, such as handling network requests (including error handling, retries, caching, and cancellation), building custom native integrations, repairing or extending faulty code in partially-correct codebases, keeping pace with the latest versions and breaking changes of common third-party libraries, and more. Future work should also explore judge-robust scoring (e.g., multi-judge consensus and calibrated partial credit) to reduce dependence on any single evaluator while maintaining transparency.
0 commit comments