Beanstalk Longshot test farm
Menu

#1873 perf(vk): a start reads the pipeline notes file once, not twice (about 0.28 s off every start)

by klimovich008 closed Open on Forgejo

Summary

The change passed on Linux NVIDIA, Linux RADV and Windows NVIDIA; these tests had no performance thresholds. On the Linux GTX 1070, field frame time p99 worsened from 23.66 to 24.88 ms (+5.2%); on RADV, p99 rose from 21.00 to 21.18 ms (+0.9%), while maximum field frame time rose from 28.26 to 67.63 ms and one field hitch was recorded. On the Windows RTX 3090, p99 was essentially unchanged at 13.63 → 13.65 ms, while load-at-create fell from 287 to 21 ms. A separate Windows RTX 4080 report failed during the baseline build before the game ran, indicating a PC-specific build problem rather than a candidate gameplay failure.

Written by AI (gpt-6.1-terra) from this commit's results, once every target finished.

Commit

Head
cefaf33
Base
main 89c3d21
Approval
approved klimovich008, · reviewed

Plan (gpt-6.1-terra)

Compare startup and Hyrule Field on main and the PR across Linux NVIDIA, Linux RADV and Windows NVIDIA

In plain words

Compare two alternating starts per side on Linux NVIDIA, Linux RADV and Windows NVIDIA PCs, then teleport from Kokiri Forest to Hyrule Field. Each tester takes about 22 minutes; startup, shader preparation, load-at-create time and gameplay frame metrics are reported without performance thresholds. A crash or failure to reach Hyrule Field within the timeout fails the test. Matching hardware must be available for each target; otherwise that target waits.

Asked for

Start-up time, this PR against main: the PR reads the pipeline notes file once a start instead of twice, about 0.28 s less on an RX 9070 XT with AMD's Windows driver. Nothing changes in play. What this PC can't show: Linux and NVIDIA. - Targets: Linux with NVIDIA, Linux with AMD (RADV), Windows with NVIDIA (`vram_gb = ">= 6"` on Windows targets), as many as are free. - One scenario from `kokiri-start`, uncapped and windowed: `frame_rate_cap = 6`, `vsync = 2`, `frame_generation = 0`, `fullscreen = 0`. `wait_s = 10` before the teleport to Hyrule Field, `timeout_s = 120` on its map wait, then measure 20 s. - 2 starts a side (main and this PR, alternating). - Probes: `boot_time` and `shader_compile` (the start's `vk: precompile:` seconds) are what this PR changes. Also a `log_value` of the load-at-create line's milliseconds (`load at create \(RX_VK_LOAD=create\).*; in (\d+) ms`), which holds main's second walk. - Pass rules: no crash or hang. `boot_time_s`, `shader_compile_s`, `fps_mean`, `frametime_p99_ms`, `run_hitch_count` and `run_frametime_max_ms` are report-only (one start's time is noisy, and play should be unchanged).

Raw plan 110 lines Download plan.json

  "pass" 
    "crash" false
  
  "probe" 
    
      "kind" "crash"
    
    
      "kind" "boot_time"
    
    
      "kind" "shader_compile"
    
    
      "kind" "frametime"
    
    
      "kind" "draws"
    
    
      "kind" "log_value"
      "value" 
        
          "name" "load_create_ms"
          "take" "max"
          "pattern" "load at create \\(RX_VK_LOAD=create\\).*; in (\\d+) ms"
        
      
    
  
  "matrix" 
  "schema" 7
  "target" 
    
      "name" "linux+nvidia"
      "tags" 
        "linux"
        "nvidia"
      
      "max_reports" 2
      "min_reports" 1
    
    
      "name" "linux+radv"
      "tags" 
        "linux"
        "radv"
      
      "max_reports" 2
      "min_reports" 1
    
    
      "name" "windows+nvidia+vram-ge6gb"
      "tags" 
        "windows"
        "nvidia"
      
      "vram_gb" ">= 6"
      "max_reports" 2
      "min_reports" 1
    
  
  "summary" "Compare startup and Hyrule Field on main and the PR across Linux NVIDIA, Linux RADV and Windows NVIDIA"
  "baseline" 
    "ref" "main"
  
  "scenario" 
    
      "env" 
      "name" "startup-field"
      "runs" 2
      "step" 
        
          "do" "wait"
          "event" "in_play"
          "timeout_s" 900
        
        
          "do" "wait_s"
          "seconds" 10
        
        
          "do" "teleport"
          "to" "HyruleField_Child"
        
        
          "do" "wait"
          "map" "HyruleField_Child"
          "event" "map"
          "timeout_s" 120
        
        
          "do" "measure"
          "name" "field"
          "warmup_s" 10
          "duration_s" 20
        
      
      "start" "save:kokiri-start"
      "matrix" true
      "settings" 
        "vsync" 2
        "fullscreen" 0
        "frame_rate_cap" 6
        "frame_generation" 0
      
    
  

Targets

TargetTierReportsStatus
linux+nvidiareviewed klimovich008 1 2 2/1 pass
linux+radvreviewed klimovich008 1 1/1 pass
windows+nvidia+vram-ge6gb 6 GB VRAM or morereviewed klimovich008 1 2 1/1 pass

Reports

ReportCommitTargetTesterResultWhen
rp_2c0bd8afb270d731cefaf33linux+nvidia jabbaxipil/Blainepass outlier
rp_e9712ff617aa30e4cefaf33windows+nvidia+vram-ge6gb Flashman4/agent-de36c5pass
rp_a723f9e4a05d5beacefaf33linux+nvidia Nerdtality/agent-0452e3pass outlier
rp_8df7ec4cbef3f6dacefaf33linux+radv icyblue8/agent-368f00pass
rp_a4df5a2c5215f9e9cefaf33windows+nvidia+vram-ge6gb CubanLegend/agent-341924error

Jobs

JobCommitTargetStateTesterStartedAttemptsEstimateNote
jb_ca048cd72903397a logcefaf33linux+radv done reporticyblue8/agent-368f001
jb_d07fe2bcef62bde0 logcefaf33windows+nvidia+vram-ge6gb done reportFlashman4/agent-de36c51
jb_b2efb1e0ee7c9234 logcefaf33linux+nvidia done runjabbaxipil/Blaine1
jb_6ec1aff1fb39e44a logcefaf33linux+nvidia done runNerdtality/agent-0452e31
jb_11719fab4c3c84bb logcefaf33windows+nvidia+vram-ge6gb done build-baseCubanLegend/agent-3419241
jb_5f3a73dd52526eb9 logcefaf33linux+radv cancelled—6 the PR was closed