Synthetic Benchmarks: No Longer Needed?

Posted: Wed May 28, 2003 12:17 am

One of the hottest debates going on the Net lately is about both NVidia (article link) and ATI (article link) cheating on 3DMark with their most recent drivers. While following this subject over the last couple of weeks started me thinking. Does anyone really care what scores they get in 3DMark anymore? My fellow NVidiot, A-KO, got me to participate in a recent debate on the Guru 3D Forums. This debate proved that some non-serious gamers do care what scores they get. Which brings me to my current editorial about Synthetic Benchmarks. After you read the article please stop by the forums and let me know what you think of the subject.

Posted: Wed May 28, 2003 1:00 am

Hate to say it, but your right Tycho.

Saying that "this is the trend of gaming to come" is bull. No one can truely predict what will be hot this year or next, and only a current game can tell you if your system can handle it or not.

Take UT2k3 for example, god, the demo was released, and all I heard were people bitchin about "my system gets over 10,000 3dmarks yet I can't play for crap, and it looks like junk!", yeah, well, tweak your system for performance in one thing, sacrifice it elsewhere. So tweak your system for gaming, and not benchmarking, and you'll be fine!

God, this type of mindset could really put an end to this whole "nvidia vs. ati" type thing, and just get people to do what's important, play the damn game Smile

Oh wait, but that's just a pipe dream.

Posted: Wed May 28, 2003 6:33 am

humm... I knew benchmarking had some catch

Posted: Wed May 28, 2003 7:19 am

I think that a synthetic benchmark (such as 3DMark) is good for 'stress'-testing certain features of your card.
For example if you really want to know how many shader-thingy can be used in a scene before performance is affected.
Or how fast your card can display loads of textures.
You can use such figures (which afaik the 'pro' version of 3Dmark allows you to examine) to see if the card is as fast as 'they' claim it to be instead of trying to guess the speed....

btw : 3dmark was originally the by-product of development for the Max Payne-'engine'/game. As a result it was a good way to judge how fast that game would run on any given pc (something which was mentioned in the manual as well)

Ultimate Fanboy Joined: 18 May 2003 Posts: 508

final reality? Very Happy

That benchmark was way ahead of it's time.

Posted: Thu Jun 05, 2003 12:37 pm

No offense but A-KO did not really understand some stuff over in that 3DGURU thread. Nor do you it seems. And please no offense intended.

3Dmarks was never about current games. Its ALWAYS been about future games. A lot of people buy a video card and keep it for a long time. More than 1 year for example. How then are they suppose to tell how that card will run future games? You can not warp to the future, get a copy of that game then warp back with the results of how cards ran that game now can you. How are they going to gauge DX9 performance today? I agree real games are the best but what games are out there today that use DX9? And that's the issue. Games that use these newer features of DX take a long time. Today we just have a few games that use DX8 effects (Pixel and vertex shadders) that the original GF3 introduced about 2 years ago! See it takes A LONG TIME for games to use these effects. Thus with out synthetic how do we gauge how these games will run? We can not go by current games. Synthetic are needed to gauge future feature performance. But then how do we know these features will be used in the exact same manner? We don't. But we know that as long as they follow the specs of DX then there is hope that it will be close...

Now the common mistakes people will make is trying to tie 3dmarks to FPS in games. You can not. But if you see two cards that are separated by 1000 3dmarks...provided no one cheats (hehehe) then in future games chances are the card is 1000 marks faster in 3dmarks will probably be faster future games.

We can see that today nV FX is painfully slow on DX9 shaders both 3Dmark2003 and independent (shadermark) benches. Does that mean that future DX9 games will run slower? Maybe its still an interesting data point that a customer today should know and if you do away will Synthetic they how we will know????

Now I really don't understand why now every one is against 3dmarks. The problem with 3dmarks are valid. But they have been valid for the very first 3dmark back in 99!!!! Why now all the fuss?

Also the game engine myth is bunk. Look at how the Kyro2 card did in 3dmark2001. It did bad and lost to the GF2Mx. Yet in MaxPayne the same engine that 3dmark2001 used the K2 was FASTER! Now consider Q3 benchmarks. People are still looking at those scores correct? But if you look at today's games based of the Q3 engines are there any similarities? No. Not at all. Q3 scores are always faster for NV cards, yet in RtCW or JK2 benchmarks (both using Q3 engine) the ATI cards are just a bit faster. And look at Splinter Cell and UT2k3. Both use the same Unreal engine. NV is faster in UT2k3 but slower in SC. So you see even in games that use the same engine you have different results cross the board. Thus card A performance in game A does not equal cards A performance in Game B even though game A and B use the same engine!!!

My point is Synthetic are valuable tool to use in review but only as part of the over all review. I would also would go with game bechnies but until they can bench every game out there and bench future games then we will need synthetic to gauge or at lest give us some in-site on how future features will run on the current hardware. With out them the todays buyer is going to have to make a choice based on a lot of un-knowns....

Jb

Posted: Thu Jun 05, 2003 1:16 pm

JB,

You bring up some valid points. However the problem is that most of the emphasis in on getting faster marks for more points. They should be concentrating on making the drivers better in the game than in a generic synthetic benchmark.

Thanks for your input and definitely for taking the time to stop by the forums and comment.

Smile

Posted: Thu Jun 05, 2003 1:38 pm

JB -
Are you still using a GF3? According to you, that's all we need right now. I don't see why anyone would you buy a video card based on games 6 -18 months in the future. By the time that game gets here... there are new video cards out that will play the game much better/faster. I haven't seen one game review this year that was tested on a GF2/3. If I was looking into the future anticipating a paticular game, I sure wouldn't buy my video card now. As you stated, you can't even tell how well a certain video card will do on a widely used game engine. Results will vary from game to game using that same engine. I base my video card buying decisions on what I'm going to use it for now, not in 1 1/2 years. Maybe I upgrade more than most, but I've never had a video card in my personal box for 12 months.

Posted: Thu Jun 05, 2003 3:22 pm

JB:

Why now all the fuss? Becuase now, unlike in 99, Graphics cards are more and more very VERY close to one another. So personally, imho, I think that the best result will still come from gaming directly.

Ok, sure, this card does better in one game, and worse in another. That's life, and the power of different programmers to use different options. However, the point remains that if you want to play "said style of game", you should benchmark your current system against that style.

Hell, I know a lot of people who play nothing more than Starcraft or similar strategy style games. What do they need the latest GF-FX or ATI 9800 Pro for? Nothing. But they bought one on the 3dMark results alone. That's a pretty lame excuse to shell out $300+ imho.

If they had benchmarked it against the games they played, they could be just fine with a GF2MX for prolly the rest of their natural life (or atleast til Windows was updated in such a way that those games would no longer run). I'm not saying 3dMark has no purpose, I'm saying that the focus of it is incorrect. Sure, it's about Future Games, but nothing, NOTHING, will tell you how the gaming industry is going to go like a CURRENT game. (Case in point, OpenGL vs. DirectX, some gamers prefer to run in 1, others in another, yet only the most current trends will tell you which you should focus your system on) (though, personally, I think you should focus on both Wink

)

As Tycho has mentioned, it's rather annoying to see people who get these majorly high 3dMarks, then complain when their system can't even run the latest games without issues. The focus is all wrong.

Posted: Thu Jun 05, 2003 4:45 pm

Tycho,
thanks you have some nice forums here.

"However the problem is that most of the emphasis in on getting faster marks for more points. They should be concentrating on making the drivers better in the game than in a generic synthetic benchmark. "

I agree that is bad. But why is that the Benchmarks fault? Doesn't the problem lie with the IHVs then? Are they forced to spend time to optimize or is it THEIR choice? Do you know if they have spent time optimizing other games just for their benchmarks scores? Again its not the benchmarks fault that other companies have to cheat and lie to us about where their priorities are. However had they made legitimate optimizations to their DX rendering pipeline then BOTH 3dmarks scores would increase as well as REAL GAME scores. That the issue here. What is a valid optimization..and what is taking it to far (dropping IQ by forcing Fp12/16 paths, using clip planes, ect)......

"JB -
Are you still using a GF3? " Nope. I got a R300 for its AA /AF as well as more fill rate/memory bandwidth.

"According to you, that's all we need right now."
No that's the tech the games are using. Please name me the games that use DX8 Pixel and vertex shaders?

"I don't see why anyone would you buy a video card based on games 6 -18 months in the future. By the time that game gets here... there are new video cards out that will play the game much better/faster. I haven't seen one game review this year that was tested on a GF2/3. If I was looking into the future anticipating a particular game, I sure wouldn't buy my video card now. As you stated, you can't even tell how well a certain video card will do on a widely used game engine. Results will vary from game to game using that same engine. I base my video card buying decisions on what I'm going to use it for now, not in 1 1/2 years. Maybe I upgrade more than most, but I've never had a video card in my personal box for 12 months."
Counter Strike (or DoD, HL, TFC, (insert any other half life mod in here) are still very popular games. According the last Value poll most of these people that play these games are using a GF2 class of video card!!!!!!!!
What's does that tell us? It tells us that some people don't upgrade very often. I upgrade about every 6 months with a new video. But I live for AA/AF at high res. People are not buying a card today for a game that comes out 12 months from now..but if they do have to get a card today and they will have to keep if for 18 months then don't they deserve to get an idea of the performance of that technology that's going to be used in that game? Again not every one upgrades as often as we do. So they will have to hold onto a card for 18 months or longer.

'Why now all the fuss? Because now, unlike in 99, Graphics cards are more and more very VERY close to one another. So personally, imho, I think that the best result will still come from gaming directly"
Actually you hade the GF2 vrs the V5 back then and they were just as close. Oh the flame wars...I still think I have burn marks from some of those Smile

Actually, today, if you look closer they are not that close. One is clearly better in DX9 test. One is clearly better in no AA/AF conditions. One is clearly better with AA only, ect...
No its only because nV's cards do so poorly that we have an issue with 3dmark2003. Notice how back when the GF3 was launched 3dmark2001 was show by nV to show of the GF3 Pixel Shaders in the Nature test the day of the GF3 launch. Perfectly fine for NV to endorse 3dmark back then as the GF3 was the ONLY card that could run the GT4 (nature and shader test). Heck they even sold the NV Performance Analyzer to other companies (that tool read the 3dmark data base and reported back the best NV card for you to get). So these same flaws with 3dmark2003 were similar to 3dmark2001. But unsteady of standing up and say whoa this is a bad bench and you folks should use real games, nV showed it off and made a bucket load of cash on 3dmark for over 2 years!!!! I'd be willing to bet that had nV cards done a lot better on 3dmark2003 they none of this would be an issue.

Peoii your missing the part. Your correct buy saying you should look to the style of games you play then get those. But until we see benchmarks on all of the games we are going to have gaps. How do we judge how one game that is not in any one of the benchmarks types will run? Guess? I don't know. But Synthetics have a valid place to help us figure out some things. They maybe useful for future games. They may not. None of us can tell for sure unless we can tell the future..if that's the case I will buy the lotto ticks and split them with you Smile

Synthetics have their place. They can tell us the fill rate, polygon performance, shader (vertex/pixel) and other useful data points. Again no one should ever place all of their trust in them. But they are a piece of the puzzle.

Jb
ChaosUT

Posted: Thu Jun 05, 2003 8:28 pm

Point of the matter is, gaps are only applicable to a gamer that plays every type of game out there.

MOST gamers have their "loves", be it FPS, RTS, whatever. Sure, one card may do 5-10fps better on one engine than on another, so WHAT?!?!?!? People complain about the wrong things, it's not "how many FPS can you crank out of it", once you hit a certain point (I don't care what you say, flame me, it's truth), you cannot tell! Heck, I saw people complaining that they got 110 in UT, and 100 in Quake3...... Seriously now, no one could tell the difference if you had the stats off. pointless I say. So as long as your card will play the type of games you want it to play, I think there really isn't a point for a synthetic benchmark.

Heck, my GeForce4 is going strong, and prolly will continue to for a while. Before this, I had a GF2MX, and it ran things just fine, only reason I got bigger and better? I wanted to be able to turn on all the "prettyness factor".

Point is, there is no one best card for everything, and no benchmark EXCEPT playing the games you want to play on that card, will ever give you any information on how good your specific card will handle those games. It's like pairing wine with food, some like it one way, some like it another, sure, there are critics out there who will tell you one thing, but you know what? Their usually wrong for my tastes 80% of the time.

So, if 20% is worth it to ya, go ahead, use the synthetic benchmarks, I'm just sayin, nothing like playing what you want to play on your system to get you the best understanding of how your system will handle it.

Posted: Fri Jun 06, 2003 1:04 am

JB,

You are correct that we need to put pressure on the IHV's first. Kyle from HardOCP is working on a proposal to do just that. I talked to Kyle yesterday and told him that if he needed help from the Unreal Community alls he has to do is ask.

I am posting some new info on this on tomorrow's news.

Wink

Posted: Fri Jun 06, 2003 10:38 am

I am sure you have seen the new info that shows remaning of 3dmarks results in higher IQ and lower Scores on the NV FX line: http://www.tech-report.com/etc/2003q2/3dmurk03/index.x?pg=1

I hope at least even though you guy may not like 3dmark cheating to inflate scores is wrong. Dumbing down IQ to increase scores is wrong. The fact that you don't like 3dmark still does not change the issue here. Many big companies use 3dmark in the suite of test (Like Dell for example) to figure out what cards to offer. Cheating is wrong and there is no defense. I am really disapointed in NV now.

Tycho,
I am not a big fan of Kyle. He ran with the ATI Q3 issue using info provided by NV. Now that the tables have turned he is completely blowing this off. That not what a good journalist would do. And thats why I stop going over there. I would rather get my info from sites that have a more balanced view on things (Tech Report, Beyond3D, ET, ect). I can also help with unreal stuff if needed.

Peoii the point is how to test future stuff? No games will exist to use those features until a long time passes. How do you gauge the DX9 prefromace on a ATI 9500+ or a GF FX card today? With out Sythetics there is no way so your making a guess.

Posted: Fri Jun 06, 2003 10:39 am

The above post was mine..maybe I should reg here Smile

Jb
ChaosUT

Posted: Fri Jun 06, 2003 10:59 am

Yes you probably should register if your going to post this much LOL. As far as Kyle is concerned yes he can be bullheaded sometimes but most of the time he means well. If he succeeds in this project that will only HELP gamers in my opinion.

I frequent Tech Report all the time ever since Damage left Ars Technica (man I am showing my 'Net age there). They definitely give things a different perspective on events. I did see the 3Dmurk article and I am still shaking my head. At this point I am not happy with ATI, NVidia or Futuremark right now. All of them are just adding to the drama and YOU and I are the ones getting hurt.

I mean I finally had some time to read the reviews of the 5600 series cards. As I was thinking of upgrading to that or a Radeon 9600 this summer. Man I am not impressed with these cards and their performance in UT2003. The only good news out of all of this is my Visiontek Ti 4200 (128) that I bought on June 30th of last year is becoming more and more of a great deal.

I really hope the industry pulls its head out of its ass soon. We have too many good games coming out to not have good cards to run it.

Crying or Very sad