Beanstalk Longshot test farm
Menu

#1955 WIP: perf(nvn): ReportCounter reads its first-report flag before the locked exchange

by siantm open Open on Forgejo

Commit

Head
38094d7
Base
main 6e1521a
Approval
approved futondestroyer, · reviewed

Plan (gpt-6.1-terra)

Compare two-minute Castle Town Market frame times and counter diagnostics against main

⚠️ Note: The farm cannot compare arbitrary counter-type log sets or enforce one line per observed type; the log probe only checks for the diagnostic's presence and marks its absence unjudged.

⚠️ Note: The 5% p99 allowance is a fixed proxy for run-to-run spread, not a statistical test of that spread. The two-minute measurement is stationary after a short movement through the Market.

In plain words

Compare the PR with main in Castle Town Market on Linux NVIDIA, Linux RADV, Windows Intel and Steam Deck. Each side gets two interleaved runs with a two-minute frame-time measurement; the plan fails on a crash or a p99 regression over 5%. The Windows Intel target will wait for a matching tester. Each tester's game runs take roughly 30 minutes. No tester online now matches windows+intel: that target waits for one.

Asked for

This makes NVN's ReportCounter read its first-report flag before the locked exchange (host/nvn_native.c), so the exchange runs only for the first report of each counter type. It has not run in the game here with this head yet. Please play Castle Town Market for 2 minutes on Linux with NVIDIA, on Linux with AMD (RADV), on Windows with an Intel GPU and on the Steam Deck, the PR against main. No crash; the log's "nvn: the game reports counter type" lines must be the same on both sides, once per type; frame time p99 equal to main's or better, within the spread.

Raw plan 117 lines Download plan.json

  "pass" 
    "crash" false
    "screenshot" 
      "max_diff" 0.05
    
    "frametime_p99_ms" 
      "max_delta_pct" 5
    
    "log_expected_missing" 
      "max" 0
    
  
  "probe" 
    
      "kind" "crash"
    
    
      "kind" "frametime"
    
    
      "kind" "draws"
    
    
      "kind" "log_match"
      "missing" "unjudged"
      "expected" 
        "nvn: the game reports counter type"
      
      "forbidden" 
    
  
  "matrix" 
  "schema" 10
  "target" 
    
      "name" "linux+nvidia"
      "tags" 
        "linux"
        "nvidia"
      
      "max_reports" 2
      "min_reports" 1
    
    
      "name" "linux+radv"
      "tags" 
        "linux"
        "radv"
      
      "max_reports" 2
      "min_reports" 1
    
    
      "name" "windows+intel"
      "tags" 
        "windows"
        "intel"
      
      "max_reports" 2
      "min_reports" 1
    
    
      "name" "steamdeck"
      "tags" 
        "steamdeck"
      
      "max_reports" 2
      "min_reports" 1
    
  
  "summary" "Compare two-minute Castle Town Market frame times and counter diagnostics against main"
  "baseline" 
    "ref" "main"
  
  "scenario" 
    
      "env" 
      "name" "castle-town-market"
      "runs" 2
      "step" 
        
          "do" "teleport"
          "to" "Market_Child"
        
        
          "do" "wait"
          "map" "Market_Child"
          "event" "map"
          "timeout_s" 60
        
        
          "x" 0.0
          "y" 1.0
          "do" "stick"
          "stick" "L"
          "frames" 180
        
        
          "do" "shot"
          "name" "market"
          "upload" false
          "compare" "baseline"
        
        
          "do" "measure"
          "name" "market"
          "warmup_s" 10
          "duration_s" 120
        
      
      "start" "save:kokiri-start"
      "matrix" true
      "settings" 
    
  

Targets

TargetTierReportsStatus
linux+nvidiareviewed futondestroyer 1 2 0/1 pass, 2 error
linux+radvreviewed futondestroyer 1 2 0/1 pass, 2 error
windows+intelreviewed futondestroyer none yet 0/1 pass, 0 running, 2 queued
steamdeckreviewed futondestroyer none yet 0/1 pass, 0 running, 2 queued

Reports

ReportCommitTargetTesterResultWhen
rp_d4ccfedbe156968138094d7linux+nvidia masterstiker46/sian-gtx1060error outlier
rp_d75027fcc9e251bf38094d7linux+radv manasuazo/agent-03253derror
rp_ce9d3a5b62de9b1b38094d7linux+nvidia Hidden testererror outlier
rp_607343a079d2dad438094d7linux+radv Simon-Olivier/agent-1b2b4ferror

Jobs

JobCommitTargetStateTesterStartedAttemptsEstimateNote
jb_1a51441befe402b0 log38094d7linux+nvidia done reportHidden tester1
jb_aeb4be00b2763604 log38094d7linux+nvidia done runmasterstiker46/sian-gtx10601
jb_3aa1f80e697e5d71 log38094d7linux+radv done build-baseSimon-Olivier/agent-1b2b4f1
jb_9106a107f68d2643 log38094d7linux+radv done reportmanasuazo/agent-03253d1
jb_d142bc5d03798c8938094d7steamdeck queued—1 No tester online can take it now.1 tester matches `steamdeck` online, but 1 ran or skipped this job already.
jb_af45da31b1b3cf8e38094d7steamdeck queued—2 No tester online can take it now.1 tester matches `steamdeck` online, but 1 ran or skipped this job already.
jb_6f81ea9c646ec66c38094d7windows+intel queued—0 No tester online can take it now.No tester online matches `windows+intel`.
jb_0b99f6936ad8184838094d7windows+intel queued—0 No tester online can take it now.No tester online matches `windows+intel`.