Samsung is clearly violating the reason of benchmarking, which is to test system performance under similar conditions a game would cause. Results from performance tests unachievable during a game are meaningless. And it's not that Samsung lets every app run at max clocks if needed until it needs to be thermally throttled down, they are specifically "boosting" benchmarking apps because it ultimately makes their phone look better.
Unfortunately, many, if not the most, big corporations tend to try such sloppy tricks on consumers every so often. Not necessarily because they are axis of evil, but rather because of unlimited pressure on worker bees from the top to produce superb results no matter what. Executives are smart, so they usually don't explicitly direct R&D and marketing to cheat on consumers, but when pressed for outstanding performance in tight time windows, engineers and product managers have limited choice. If the trick is busted, executives can deny their involvement, and engineers will be scapegoated, if needed. This is the side effect of bad capitalism and profit maximization.
"... many, if not the most, big corporations tend to try such sloppy tricks..."
In a word, "BULL!"
The vast majority of corporations do not play such shady games.The vast majority do not setup -- and keep hidden until caught -- one set of capabilities for the vast majority of uses while utilizing a different set of capabilities for benchmarks and their own applications.
By saying "everyone's doing it" you seem to be saying "Everyone does it so it's OK to do." It's not and never should be.
Yes, it does happen. (I remember such things as far back as the Dhrystone fiasco years ago.) However, every time it comes to light the perpetrators should be treated very, very harshly. No one should have to stand for this crap.
If my posting offended you, I'm sorry, but seriously I don't understand how you could interpret it such a way. I suggest you to take Reading Comprehension 101 at a local community college.
Nice... attack someones character just because you don't agree with his statement. Maybe you should take a class at YOUR local community college in Debate to realize that turning to character-assination in a reply to an argument is the first sign of a weak and pathetic argument. SEE how easy that is
nvidia and apple both got caught doing similar benchmark boosting, that 2 big corporations that i can immediately recall, I'm sure you can google a bit and find out more.
And they should have been castigated and publicly exposed for doing so. Now its Samsung and so we need to expose this and get them to cease this deplorable activity.
You're being extremely naive. The vast majority of companies HAVE done this AT SOME IN TIME, most of the time until they were caught. Key word: caught. Companies to this day design their hardware around benchmarks, or they might even attempt to release their own benchmarking software.
I'd bet my sister that most are doing this in far more subtle and far more undetectable ways.
So your saying because there device is programmed to not use it's GPU to it's full potential when it's not needed is a bad thing? Are you that dense? It's not to make there device look better, it's to safe the user battery life.
"Under ordinary conditions, the GALAXY S4 has been designed to allow a maximum GPU frequency of 533MHz." This is absolute bullocks. CLEARLY after these two postings / findings by Anand and Brian, they are cherry-picking which Apps of access to this higher "boost-mode". This might as well be called "cheat-mode" because it is not something all apps can use if the thermal and battery requirements are there, and it seems to target mostly benchmarking apps (i.e. GLBenchmark, Quadrant, and AnTuTu) and superfluous apps such as their camera app, gallery app, and video player, mostly because most people wouldn't test to see if there was any shenanigans going on in these stock apps.
Kudos to AnandTech for finding these shandy engineering tactics and exposing them to the general public. I don't care which company has done this, if a company is engineering ways to cheat / boost their results on benchmark tests, they need to be ridiculed and shamed regardless of how prevalent it is. And if this company is cherry-picking only certain apps from having access to this type of higher GPU frequency-mode, then they should be even more ridiculed and shamed.
Don't forget that the switching from the A7 cores to the A15 cores is also not instantaneous. The normal thing would be for the increased load from a benchmark app to eventually push the device to its power-hungry A15 cores. However, Samsung locked the device into using those CPU cores form the get-go and refused to let it drop to the A7 cores.
The big problem with ARM A15 cores is that they were designed for higher power consumption and not for mobile. The Big Little config with the A7 cores is there to mitigate that and it is part of the real-world performance profile of the device. Samsung has made it so the CPU benchmarks strictly use the A15 cores, which is very misleading (In addition to over-clocking the GPU).
Turbo doesn't take input from specific applications/OS calls. Intel's CPU's do not increase their clock speeds when a specific application is run.Thermal level, power consumption and thread load are the three metrics which determine what frequency the chip runs at with tubro.
The difference is that turbo boost can potentially be used by every application. It basically says "If you have thermal headroom, use the boost." And it is done in hardware. Samsung on the other hand only allows that boost for benchmarks, which is bad. And they should feel bad...
Samsung is never going to admit wrong doing. To do so loses face. They would have to admit they are cheating. They have less shame than Lance Armstrong. So Samsung is never going to admit wrongdoing.
I guess you missed the single MOST IMPORTANT point of the post above: "Furthermore, I think this also highlights a real issue with the way DVFS management is done presently in the mobile space. Software is absolutely the wrong place to handle DVFS, it needs to be done in hardware."
Doing fine-grained power management scaling in HW is absolutely the correct way to do things. It is more responsive, it allows access to more information about whatever the relevant parameters are (current budget, thermals,...), it can't be borked by a broken driver, etc etc.
It will be interesting (though will we ever know?) to see how rapidly Apple goes down this path. They have the total control which allows them to decide whether they want to make the tradeoff in HW vs SW, and my guess is they will want to move to HW ASAP (gated, of course, by their current practical concerns --- getting to 64bit soon, and figuring out where the thing will be fabbed and what that implies going forward).
Seems like Samsung is trying to maintain a whitelist of apps that are almost all GPU, minimal CPU, and and staying inside the thermal envelope.
They somehow can't get proper cache coherency in this weird big.little octa-core beast, so they have to do gonzo tricks in software to deliver a stable platform.
Actually maybe they are being conservative with this thing. But it's the best they can do right now.
It seems to be an engineering thing, not a cheating-on-tests thing.
Rezurecta, you should perhaps change that comment to say "theoretical peformance within thermal constraints" and then it would be more accurate. Professionaly written 3D graphics benchmarks like the one being discussed here exist to serve as a predictor of the relative performance for real world applications like games or in some caes GUI. Games are seldom a practical way for reviewers like AnandTech to quantify the relative performance of one platform vs. another...note I say "quantify", as in having a frame rate number that is repeatable irrespective of who is doing the testing, and reported numerically in a manner that can be compared from one platform to the next. You are correct that games should ALSO be tested to round out the analysis (noting at least the qualitative results), but Kishonti's GLBenchmark tests are a reasonable and repeatable proxy for actual game testing. So too are the graphics benchmark tests from Rightware, Futuremark and others. Antutu's so-called graphics tests are a joke by comparison today, but I'm told that it will get better in a future future release. Detecting a gaming style benchmark like GLBenchmark and invoking a GPU clocking scenario that is unattainable for sustained use games just to obtain a benchmark higher score is cheating. I think everybody can understand why that's the case. Therefore Samsung should amend their statement to read as follows: "Under ordinary conditions, the GALAXY S4 Exynos 5410 has been designed to run at a GPU frequency of 480MHz. However, the maximum GPU frequency is rasied to to 532MHz primarly for benchmarks but also for any application where the graphics rendering workload is substantial only for a period of a few seconds or less." Another form of cheating on benchmarks is if the workload of a benchmark itself is changed through application detection software, with an "optimization" that would not be applied for a similar scenario with a real world application. Kudos to AnandTech for exposing both of them. http://forums.anandtech.com/showthread.php?t=23300...
Clearly it is not safe for samsung to allow users/developers to access the higher frequency that would demand higher power and generate higher heat. A rogue behavior from a bug, intentional or not, could make it a safety issue for a handheld device to get overheated real quick and then who would be blamed? the developer? It's gonna be samsung's ass on the line and I'd side with them to be safe than sorry.
But even a 10% boost in several benchmarks, if the reason for it isn't known, can be enough to move a device to the top of the chart. And there will always be enough people to think that it's a good reason to buy the product.
And then, of course, that top rating will be written about, and even used in promotional materials, etc. overall, it's simply not proper, and somewhat slimey
So they're claiming that they're not boosting clock speeds for higher benchmarks, but it just happens that those benchmarks are given access to the maximum GPU clock speed but other non-Samsung apps aren't. Yeah, somehow that comes off as no less shady.
Shame on you Shamesung! This is clearly cheating... and there should be consequences. And they even lie about it, as you guys clearly showed in this Update.
In relation to the camera app running at full frequency for filters, this might be a strong pointer to samsung using gpgpu compute for those filters. Imagination did specifically demo such filters taking significantly less power and running quicker when done on the gpu rather than the CPU, and did indicate that licencees would be using the gpgpu computer facilities, in some interviews done a while before the samsung chip/phone was released.
If the boost is not available outside (selected) benchmarks there is absolutely no point in having it - except cheating.
Clever power management, on the other hand, is something phones really need. As far as I know there are already different power settings in Android. To make actual use of boost all one would have to do is to include it there for the highest setting. Offer separate settings for "on battery" and "at the wall" (don't know if this is already there) and check SoC temperatures - reduce maximum voltage and frequency bins if too hot. If done correctly it's Intel Turbo done in software.
Well... Considered how other devices are heavily criticised and have negative media impact because of a conservative approach (ie thermal throttling in N4) I would have done the same.
Bored people are happy with big numbers in the benchmarks? Let them be. No one can't make the difference between 450 and 500 MHz in real life usage so who fucking cares.
So let's say samsung allows a new demanding 3d game to run at the 533 mhz speed, does that mean they stopped cheating? or they are cheating more now because now they cheat the benchmark and the game? By the same logic anyone who doesn't use demanding apps like games are being cheated 100% of the time regardless of whatever devices they are using
Your original finding was that ONLY benchmarks accessed the highest speed, that's quite a bold claim, since that meant that you stress tested ALL apps available to the galaxy s4. Then you found out that at least one other app, the camera, also accessed the highest speed. so shouldn't you retract your original statement?
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
38 Comments
Back to Article
jeffkibuule - Wednesday, July 31, 2013 - link
Samsung is clearly violating the reason of benchmarking, which is to test system performance under similar conditions a game would cause. Results from performance tests unachievable during a game are meaningless. And it's not that Samsung lets every app run at max clocks if needed until it needs to be thermally throttled down, they are specifically "boosting" benchmarking apps because it ultimately makes their phone look better.It's just some shady stuff.
twotwotwo - Wednesday, July 31, 2013 - link
Right--the frequency cap may have made sense, but if so it ought to be on everywhere.MKBL - Wednesday, July 31, 2013 - link
Unfortunately, many, if not the most, big corporations tend to try such sloppy tricks on consumers every so often. Not necessarily because they are axis of evil, but rather because of unlimited pressure on worker bees from the top to produce superb results no matter what. Executives are smart, so they usually don't explicitly direct R&D and marketing to cheat on consumers, but when pressed for outstanding performance in tight time windows, engineers and product managers have limited choice. If the trick is busted, executives can deny their involvement, and engineers will be scapegoated, if needed. This is the side effect of bad capitalism and profit maximization.Shadowself - Thursday, August 1, 2013 - link
"... many, if not the most, big corporations tend to try such sloppy tricks..."In a word, "BULL!"
The vast majority of corporations do not play such shady games.The vast majority do not setup -- and keep hidden until caught -- one set of capabilities for the vast majority of uses while utilizing a different set of capabilities for benchmarks and their own applications.
By saying "everyone's doing it" you seem to be saying "Everyone does it so it's OK to do." It's not and never should be.
Yes, it does happen. (I remember such things as far back as the Dhrystone fiasco years ago.) However, every time it comes to light the perpetrators should be treated very, very harshly. No one should have to stand for this crap.
MKBL - Thursday, August 1, 2013 - link
If my posting offended you, I'm sorry, but seriously I don't understand how you could interpret it such a way. I suggest you to take Reading Comprehension 101 at a local community college.Dman23 - Thursday, August 1, 2013 - link
Nice... attack someones character just because you don't agree with his statement. Maybe you should take a class at YOUR local community college in Debate to realize that turning to character-assination in a reply to an argument is the first sign of a weak and pathetic argument. SEE how easy that isRadarTheKat - Sunday, August 4, 2013 - link
I suggest MKBL also consider an Ethics class.chuchurocket - Saturday, August 3, 2013 - link
nvidia and apple both got caught doing similar benchmark boosting, that 2 big corporations that i can immediately recall, I'm sure you can google a bit and find out more.RadarTheKat - Sunday, August 4, 2013 - link
And they should have been castigated and publicly exposed for doing so. Now its Samsung and so we need to expose this and get them to cease this deplorable activity.edward kuebler - Tuesday, October 1, 2013 - link
Apple? When did that happen?jleach1 - Wednesday, August 7, 2013 - link
You're being extremely naive. The vast majority of companies HAVE done this AT SOME IN TIME, most of the time until they were caught. Key word: caught. Companies to this day design their hardware around benchmarks, or they might even attempt to release their own benchmarking software.I'd bet my sister that most are doing this in far more subtle and far more undetectable ways.
Kirus93x - Thursday, September 19, 2013 - link
So your saying because there device is programmed to not use it's GPU to it's full potential when it's not needed is a bad thing? Are you that dense? It's not to make there device look better, it's to safe the user battery life.Krysto - Wednesday, July 31, 2013 - link
Isn't what Samsung is doing pretty much what Intel is doing with the Turbo-Boost activated in benchmarks, too?BillyONeal - Wednesday, July 31, 2013 - link
No, turbo does not favor anyone. It just allows the CPU to use unused thermal headroom.Dman23 - Wednesday, July 31, 2013 - link
"Under ordinary conditions, the GALAXY S4 has been designed to allow a maximum GPU frequency of 533MHz." This is absolute bullocks. CLEARLY after these two postings / findings by Anand and Brian, they are cherry-picking which Apps of access to this higher "boost-mode". This might as well be called "cheat-mode" because it is not something all apps can use if the thermal and battery requirements are there, and it seems to target mostly benchmarking apps (i.e. GLBenchmark, Quadrant, and AnTuTu) and superfluous apps such as their camera app, gallery app, and video player, mostly because most people wouldn't test to see if there was any shenanigans going on in these stock apps.Kudos to AnandTech for finding these shandy engineering tactics and exposing them to the general public. I don't care which company has done this, if a company is engineering ways to cheat / boost their results on benchmark tests, they need to be ridiculed and shamed regardless of how prevalent it is. And if this company is cherry-picking only certain apps from having access to this type of higher GPU frequency-mode, then they should be even more ridiculed and shamed.
BC2009 - Thursday, August 1, 2013 - link
Don't forget that the switching from the A7 cores to the A15 cores is also not instantaneous. The normal thing would be for the increased load from a benchmark app to eventually push the device to its power-hungry A15 cores. However, Samsung locked the device into using those CPU cores form the get-go and refused to let it drop to the A7 cores.The big problem with ARM A15 cores is that they were designed for higher power consumption and not for mobile. The Big Little config with the A7 cores is there to mitigate that and it is part of the real-world performance profile of the device. Samsung has made it so the CPU benchmarks strictly use the A15 cores, which is very misleading (In addition to over-clocking the GPU).
Kevin G - Wednesday, July 31, 2013 - link
Turbo doesn't take input from specific applications/OS calls. Intel's CPU's do not increase their clock speeds when a specific application is run.Thermal level, power consumption and thread load are the three metrics which determine what frequency the chip runs at with tubro.Chloiber - Wednesday, July 31, 2013 - link
The difference is that turbo boost can potentially be used by every application. It basically says "If you have thermal headroom, use the boost." And it is done in hardware.Samsung on the other hand only allows that boost for benchmarks, which is bad.
And they should feel bad...
jameskatt - Thursday, August 1, 2013 - link
Samsung is never going to admit wrong doing. To do so loses face. They would have to admit they are cheating. They have less shame than Lance Armstrong. So Samsung is never going to admit wrongdoing.name99 - Wednesday, July 31, 2013 - link
I guess you missed the single MOST IMPORTANT point of the post above:"Furthermore, I think this also highlights a real issue with the way DVFS management is done presently in the mobile space. Software is absolutely the wrong place to handle DVFS, it needs to be done in hardware."
Doing fine-grained power management scaling in HW is absolutely the correct way to do things. It is more responsive, it allows access to more information about whatever the relevant parameters are (current budget, thermals,...), it can't be borked by a broken driver, etc etc.
It will be interesting (though will we ever know?) to see how rapidly Apple goes down this path. They have the total control which allows them to decide whether they want to make the tradeoff in HW vs SW, and my guess is they will want to move to HW ASAP (gated, of course, by their current practical concerns --- getting to 64bit soon, and figuring out where the thing will be fabbed and what that implies going forward).
watersb - Thursday, August 1, 2013 - link
Agreed.Seems like Samsung is trying to maintain a whitelist of apps that are almost all GPU, minimal CPU, and and staying inside the thermal envelope.
They somehow can't get proper cache coherency in this weird big.little octa-core beast, so they have to do gonzo tricks in software to deliver a stable platform.
Actually maybe they are being conservative with this thing. But it's the best they can do right now.
It seems to be an engineering thing, not a cheating-on-tests thing.
Rezurecta - Wednesday, July 31, 2013 - link
Wrong, benchmarks are a measure of theoretical performance, and, as we see with PC, the only way to test gaming performance is to test the game.bronze_elephant - Wednesday, July 31, 2013 - link
Rezurecta, you should perhaps change that comment to say "theoretical peformance within thermal constraints" and then it would be more accurate. Professionaly written 3D graphics benchmarks like the one being discussed here exist to serve as a predictor of the relative performance for real world applications like games or in some caes GUI. Games are seldom a practical way for reviewers like AnandTech to quantify the relative performance of one platform vs. another...note I say "quantify", as in having a frame rate number that is repeatable irrespective of who is doing the testing, and reported numerically in a manner that can be compared from one platform to the next. You are correct that games should ALSO be tested to round out the analysis (noting at least the qualitative results), but Kishonti's GLBenchmark tests are a reasonable and repeatable proxy for actual game testing. So too are the graphics benchmark tests from Rightware, Futuremark and others. Antutu's so-called graphics tests are a joke by comparison today, but I'm told that it will get better in a future future release.Detecting a gaming style benchmark like GLBenchmark and invoking a GPU clocking scenario that is unattainable for sustained use games just to obtain a benchmark higher score is cheating. I think everybody can understand why that's the case. Therefore Samsung should amend their statement to read as follows:
"Under ordinary conditions, the GALAXY S4 Exynos 5410 has been designed to run at a GPU frequency of 480MHz. However, the maximum GPU frequency is rasied to to 532MHz primarly for benchmarks but also for any application where the graphics rendering workload is substantial only for a period of a few seconds or less."
Another form of cheating on benchmarks is if the workload of a benchmark itself is changed through application detection software, with an "optimization" that would not be applied for a similar scenario with a real world application. Kudos to AnandTech for exposing both of them.
http://forums.anandtech.com/showthread.php?t=23300...
SteelRing - Wednesday, July 31, 2013 - link
Clearly it is not safe for samsung to allow users/developers to access the higher frequency that would demand higher power and generate higher heat. A rogue behavior from a bug, intentional or not, could make it a safety issue for a handheld device to get overheated real quick and then who would be blamed? the developer? It's gonna be samsung's ass on the line and I'd side with them to be safe than sorry.doobydoo - Wednesday, July 31, 2013 - link
The point being that if that performance isn't safe to utilise, it shouldn't be the one which is benchmarked.HighTech4US - Wednesday, July 31, 2013 - link
BINGO we have a winner/melgross - Wednesday, July 31, 2013 - link
But even a 10% boost in several benchmarks, if the reason for it isn't known, can be enough to move a device to the top of the chart. And there will always be enough people to think that it's a good reason to buy the product.And then, of course, that top rating will be written about, and even used in promotional materials, etc. overall, it's simply not proper, and somewhat slimey
James5mith - Wednesday, July 31, 2013 - link
Much like their choice of plastic for the back of the phone!(couldn't resist)
GokieKS - Wednesday, July 31, 2013 - link
So they're claiming that they're not boosting clock speeds for higher benchmarks, but it just happens that those benchmarks are given access to the maximum GPU clock speed but other non-Samsung apps aren't. Yeah, somehow that comes off as no less shady.milans - Wednesday, July 31, 2013 - link
Shame on you Shamesung! This is clearly cheating... and there should be consequences. And they even lie about it, as you guys clearly showed in this Update.Tangey - Wednesday, July 31, 2013 - link
In relation to the camera app running at full frequency for filters, this might be a strong pointer to samsung using gpgpu compute for those filters. Imagination did specifically demo such filters taking significantly less power and running quicker when done on the gpu rather than the CPU, and did indicate that licencees would be using the gpgpu computer facilities, in some interviews done a while before the samsung chip/phone was released.cbf - Wednesday, July 31, 2013 - link
So in other words, the only apps that are actually allowed to access the highest GPU speed for more than a brief period of time are benchmarks.ananduser - Wednesday, July 31, 2013 - link
Nice detective work. Let's see if wp or ios are equally at fault.BC2009 - Thursday, August 1, 2013 - link
Anand, here's a question for you....Can you deactivate the benchmark booster and re-run the Galaxy S4 benchmarks against the HTC One and iPhone 5. Would love to see the results.
MrSpadge - Thursday, August 1, 2013 - link
If the boost is not available outside (selected) benchmarks there is absolutely no point in having it - except cheating.Clever power management, on the other hand, is something phones really need. As far as I know there are already different power settings in Android. To make actual use of boost all one would have to do is to include it there for the highest setting. Offer separate settings for "on battery" and "at the wall" (don't know if this is already there) and check SoC temperatures - reduce maximum voltage and frequency bins if too hot. If done correctly it's Intel Turbo done in software.
SirPerro - Friday, August 2, 2013 - link
Well... Considered how other devices are heavily criticised and have negative media impact because of a conservative approach (ie thermal throttling in N4) I would have done the same.Bored people are happy with big numbers in the benchmarks? Let them be. No one can't make the difference between 450 and 500 MHz in real life usage so who fucking cares.
chuchurocket - Saturday, August 3, 2013 - link
So let's say samsung allows a new demanding 3d game to run at the 533 mhz speed, does that mean they stopped cheating? or they are cheating more now because now they cheat the benchmark and the game? By the same logic anyone who doesn't use demanding apps like games are being cheated 100% of the time regardless of whatever devices they are usingchuchurocket - Saturday, August 3, 2013 - link
Your original finding was that ONLY benchmarks accessed the highest speed, that's quite a bold claim, since that meant that you stress tested ALL apps available to the galaxy s4. Then you found out that at least one other app, the camera, also accessed the highest speed. so shouldn't you retract your original statement?