Beanstalk Longshot test farm
Menu

#1372 WIP: feat(perf): sample performance every second and name the client

by gab9281 open Open on Forgejo

Commit

Head
0cece02
Base
main 544229a
Approval
approved gab9281, · reviewed

Plan (gpt-6.1-terra)

Compare six minutes of Hyrule Field frame times on Linux and Windows; telemetry opt-in and summary cannot be checked.

In plain words

On Linux and Windows, this plan compares frame time and 1% low frame time in Hyrule Field for six minutes per build, after the same brief movement on each. It fails on a crash or a frame-time regression above 10%. The farm cannot enable the launcher’s performance-statistics opt-in or inspect the written summary, so sampling overhead and the requested sample count and client field remain unverified; the six-minute measurement is stationary.

Asked for

Linux and Windows, any GPU: six minutes in Hyrule Field with Send performance statistics on, the same route on main and on this PR. Compare frame time and 1% lows, to see whether sampling every second costs anything, and check that a performance summary with about 300 samples and "client":"Longshot" is written.

Raw plan 78 lines Download plan.json

  "pass" 
    "crash" false
    "frametime_p99_ms" 
      "max_delta_pct" 10
    
    "frametime_1pct_low_ms" 
      "max_delta_pct" 10
    
  
  "probe" 
    
      "kind" "crash"
    
    
      "kind" "frametime"
    
  
  "matrix" 
  "schema" 3
  "target" 
    
      "name" "linux"
      "tags" 
        "linux"
      
      "max_reports" 2
      "min_reports" 1
    
    
      "name" "windows"
      "tags" 
        "windows"
      
      "max_reports" 2
      "min_reports" 1
    
  
  "summary" "Compare six minutes of Hyrule Field frame times on Linux and Windows; telemetry opt-in and summary cannot be checked."
  "baseline" 
    "ref" "main"
  
  "scenario" 
    
      "env" 
      "name" "field-frame-time"
      "runs" 1
      "step" 
        
          "do" "teleport"
          "to" "HyruleField_Child"
        
        
          "do" "wait"
          "map" "HyruleField_Child"
          "event" "map"
          "timeout_s" 60
        
        
          "x" 0.0
          "y" 1.0
          "do" "stick"
          "stick" "L"
          "frames" 300
        
        
          "do" "measure"
          "name" "field"
          "warmup_s" 5
          "duration_s" 360
        
      
      "start" "new"
      "matrix" true
      "settings" 
    
  

Targets

TargetTierReportsStatus
linuxreviewed gab9281 1 2 1/1 fail
windowsreviewed gab9281 1 2 1/1 fail

Reports

ReportCommitTargetTesterResultWhen
rp_1438243e52d546a90cece02windows futondestroyer/windows-rtx4070pass outlier
rp_639e77a7697c776a0cece02linux jabbaxipil/Blainefail outlier
rp_905572fccfb327cd0cece02windows Hidden testerfail outlier
rp_4e04fd72f569e6ee0cece02linux Grimlock456/agent-4835b5pass outlier

Jobs

JobCommitTargetStateTesterStartedAttemptsEstimateNote
jb_f20d735e13866ee1 log0cece02linux done reportGrimlock456/agent-4835b51
jb_cbb47d6617239850 log0cece02windows done runfutondestroyer/windows-rtx40701
jb_51a6faa1687f0765 log0cece02linux done runjabbaxipil/Blaine1
jb_8dcc8155c720cb1a log0cece02windows done runHidden tester1