AFAIK, It’s not an issue of automated testing, and I don’t believe they re-test all their cards on Windows with every new review either. Instead, they maintain the same versions of software on Windows as well until enough time has passed and enough updates have piled up that they do finally re-test everything with new games to create a new dataset to compare against. They’re trying to do the same methodology on Linux.
Instead, they maintain the same versions of software on Windows as well until enough time has passed and enough updates have piled up that they do finally re-test everything
I’m not that involved with their testing procedure but doesn’t that put newer cards at a disadvantage?
They lack any sort of driver optimization if the release drivers are never installed.
That’s a good point. I went back to the video to rewatch it, and turns out I totally missed where they said they only freeze things during a testing phase, then unfreeze it after they’re done and allow updates to commence as normal.
They mentioned that due to Linux receiving more frequent updates often with meaningful performance improvements, they’ll have to throw away older data and re-test more often on Linux, as Windows doesn’t really change much in performance between updates. So I would guess that they would use release drivers with new cards, and likely would only re-test their entire suite if the release driver also gave a big performance boost on older cards.
That makes sense.
So the best option seems to be to note updates for newer cards down until the automated testing can be done on Linux as well.
AFAIK, It’s not an issue of automated testing, and I don’t believe they re-test all their cards on Windows with every new review either. Instead, they maintain the same versions of software on Windows as well until enough time has passed and enough updates have piled up that they do finally re-test everything with new games to create a new dataset to compare against. They’re trying to do the same methodology on Linux.
I’m not that involved with their testing procedure but doesn’t that put newer cards at a disadvantage?
They lack any sort of driver optimization if the release drivers are never installed.
That’s a good point. I went back to the video to rewatch it, and turns out I totally missed where they said they only freeze things during a testing phase, then unfreeze it after they’re done and allow updates to commence as normal.
They mentioned that due to Linux receiving more frequent updates often with meaningful performance improvements, they’ll have to throw away older data and re-test more often on Linux, as Windows doesn’t really change much in performance between updates. So I would guess that they would use release drivers with new cards, and likely would only re-test their entire suite if the release driver also gave a big performance boost on older cards.