android-benchmark-comparison

Android benchmark comparison

Core principle

Treat a physical Android configuration comparison as a reproducible experiment: verify comparable workloads and device conditions before interpreting a ranking or choosing a default.

Procedure

1. State the decision, configurations, workloads, metric definitions, and repetitions. Preserve exact build identity, configuration, raw results, and traces; then verify every intended case and iteration ran. Distinguish missing, failed, and excluded runs; do not compare only the fastest survivors.
2. Control and record relevant device conditions, including device model and state, thermal and power mode, display brightness, background load, and network or input conditions. Keep device-specific commands and CPU masks in the project's runbook.
3. Balance or reverse run order and repeat the comparison. Report the spread and whether the ordering holds; do not discard slow iterations after seeing the result.
4. When rankings reverse or variability is material, defer a firm default decision until the reversal is resolved. Give the complete next comparison, not just its first blocker: confirm the same named cases and iterations, repeat with balanced or reversed run order, and inspect trace data whose timestamps overlap each measured interval for placement, contention, or thermal changes. Fixed-performance mode does not prove CPU placement. If an affinity experiment was attempted, discover the device topology and verify placement during the measured interval. Restore the recorded original affinity settings after the experiment and verify that restoration before another run; do not merely note that restoration needs checking. Label verified affinity runs as controlled comparisons. If the original settings or any other check are unavailable, state the gap and keep any default choice explicitly provisional.
5. Calculate summaries from unrounded observations, then round only for presentation. Name the aggregation explicitly: the mean of per-run percentiles is not a percentile of pooled observations. Choose an aggregation that answers the stated decision; do not prescribe one statistic universally.
6. Separate controlled-experiment evidence from normal user performance. If several conditions changed together, report the comparison as more controlled but do not attribute its whole difference to one control. Use CPU frame-duration evidence to inform a visual quality/performance decision, without claiming it measures GPU shader time.
7. Finish with the raw-evidence location, completed-case counts, variability, trace findings, controls and restoration status, plus the bounded decision or remaining uncertainty. When a reversal is unresolved, state the full sequence still needed: matching coverage, balanced or reversed order, measured-interval trace inspection, and restoration of any changed affinity settings. If fixed-performance mode or an affinity command is cited, state that neither proves CPU placement without trace verification. Name run order explicitly in the recommendation: a "balanced comparison" alone does not tell the team to balance or reverse run order. Label any earlier default choice provisional.

Boundaries

• A single stable benchmark run can support a narrow observation, but not a robust configuration ranking.
• Do not turn a device-specific CPU mask, brightness value, iteration count, or summary statistic into a permanent default.
• When traces or repeat coverage cannot resolve a reversal, keep the default unchanged or make a provisional decision with that limitation explicit.