If the tool is open source, one could create a custom build with enough changes to prevent their cheat software from detecting it. In the case of PCMark, thats basically what the anonymised version is that was used here.
I think one way to change benchmarking phones isn't just to run the benchmark a few times, but to run it until battery is drained, and also publish the battery life. Obviously there isn't time to do that for every benchmark, but maybe a couple would suffice to show there is a cheating issue.
I.e., a phone bumping performance for benchmarks will have a lower battery life than one that isn't, and this will expose it.
PCMark already is a battery test in itself so they've already shot themselves in the foot here in this instance, and we've already hammered devices which cheat GPU with thermals, either by the phones getting incredibly hot or having issues with long-term performance.
True, but pychobriggsy has a great idea. For the benchmarks that are not all-encompassing, like GeekBench, we could fully charge the phone and then run the benchmark repeatedly. Then at the end, tally up how many runs it managed. What the average score was. How much the performance declined. The standard deviation. And even add all the bechmark results up for a Total Score.
Obviously, the devices which don't overclock much, they're going to have a much higher Total Score, as they would last much longer and have more benchmarks completed. Not to mention, the standard deviation and the performance drop will be much less, showing a nicer consistent result.
I would also record device temperatures and chart that during this period. In many ways the temperature of the device could be more important than battery longevity in these sorts of tests, to see if the phone becomes too hot to physically hold it in the hand while using it, which sorta defeats the point of a phone if it becomes uncomfortable to hold it in operation.
Thanks for digging into things like this guys. This will most likely not be the last of these cases that you run into. MediaTek's response was certainly disappointing.
Yeah, not the last case and not the first one too. I read that even Snapdragon 855 and Snapdragon 730G chipsets were accused of such practices. Don't consider this a cheating for real.
Benchmarks of all kinds--certainly not just mobile and certainly not just MTK--have been used by companies to cheat in many ways. (I'll probably carry the infamous nVidia GPU cheat of 3dMark years ago to my grave...;)) The solution? Ignore the benchmarks--well, if you must, then give them a glance, but *do not buy* based on benchmarks. Buy based on the track-record of the manufacturer--buy based on your experience with a certain manufacturer's products--buy based on the features you need and the money you want to spend. Speaking of bogus benchmarks: Userbenchmark, anyone?...;)
Theres nothing wrong with some benchmark skepticism, especially for something like a phone where pure performance might not be a good metric for what a customer wants.
But buying based on brand reputation and anectodal evidence isn't much of an improvement. This is doubly true for more utilitarian devices like PC hardware.
I second this. Some phones in the mid-range has a decent GPU, others don't. Benchmarks are able to bring that to light; vendors will not do it by themselves.
as an owner of note 8 pro I am dissapointed. I have it for a ~year now, its good device. And I am really sad to see they resolved to cheating.... I took it because it was cheapest usb-c equipped device with fast charge and big battery. But I did look at antutu result, and it was a factor in buying it. and now I just cannot say its good, even if my experiance is good.....
So that is why iPhone is SO MUCH better in Web tests while being less better :) in synthetics? ;) And you all thought it is because of some magic Javascript engine...
No, it's because the iPhone can afford to make a loss on their SoC and is therefore can ship with a larger CPU, plus have a Js engine that is directly compiled for the SoC while e.g. Chrome JS engine has to be general.
I think ios even beats Mac in JS tests, would have to check again though.
I like how you cooked up a baseless conspiracy theory all on your own at the beginning of the comment and by the end of it had already resorted to admonishing everyone for not believing in it like it's a fact. Incredible gaslighting skills right there.
Thanks Andrei, this and similar articles are key reasons why I come here! Three questions, one suggestion, one comment Did you or anyone here ever had a phone "fry" while running benchmarks, especially repeated loops while connected to wall power? In other words, will some of these SoC ignore the thermal shutdown, and bake themselves to death? Lastly, does the cheating extend to misreporting the SoC temperature in any of these? Suggestion: I know you do that already, but please make even more noise about the thermal performance of a phone, and add FliR measurements to whatever the SoC provides on internal temperatures. Other suggestion: Always lead the graphs and data in reviews with the "warm" mode data; the current focus on maximum burst performance with a cold SoC encourages this kind of cheating.
Last, but not least, thanks for always including the performance/Wh numbers; those are the ones I look for to evaluate whether an SoC is really performant, or artificially juiced up.
1) I've had several devices in the past thermally shut down due to overheating on a cheating benchmark. I haven't had any actually "fry" themselves as the shutdown prevented that - that'd be a whole can of worms beyond what we ever saw. Also no long-term damage.
2) I actually don't really use reported SoC temperatures anyhow because the sensors will always differ between devices. I use an IR thermometer to measure skin temperatures and do report these when notable.
3) The GPU data already is sorted with the phones sustained performance metrics. The efficiency data is more interesting at peak as it's supposed to be an analysis of the SoC, not device performance. Regular CPU loads don't actually thermally load the SoC sufficiently to actually throttle in most scenarios.
Thanks Andrei! Did any of the MTK devices go into thermal shutdown when they were stressed in "sport mode"? Also, could you take and report your temperature measurements for every device you review, not just for those that get toasty? That would help us readers put a phone's thermal performance into perspective, and reward those manufacturers that don't cheat and design their phones well. Thanks!
MediaTek SoCs don't get hot enough to overheat or throttle much anyway, the D1000 is a perfectly good chip. I have nothing against their hardware designs - this piece is all just about the software aspect of things.
I certainly agree that their new D1000 SoC looks like a real contender, and the absence of the cheat in their software for that SoC is a good sign. With that chip having 4 A77 cores as its Big core lineup, it's difficult to imagine just how hard MTK must have driven its Helio P95 SoC to surpass the D1000 in benchmark tests. On the other hand, that result apparently made you take a close look at their software, so the D1000 already brought MTK a lot of attention, even if it's not the kind they had in mind.
This is one of many reasons I am working on my own custom benchmark suite. Sometimes NOT being a well known benchmark has many benefits.
On the one hand, I can understand their argument. Mobile operating systems are optimized for power, and the chip may be constrained in unexpected ways because of this. However, their lack of transparency makes this argument sketchy at best.
Not trying to defend MediaTek of any ill-intention around the benchmark score boost, but the blame shoudn’t be just on OEMs or silicon vendors. Benchmark companies are trying to make a great deal of money out of it (ripping off SoC/OEM vendors) for so-called early access, optimization, etc. And why do SoC/OEM vendors get into the trap, knowing the not-so-healthy intention in the background? Because the benchmark scores are quoted and referenced by tech media such as yourself, and it resonates with the market & industry. I’m not trying to make an argument that you’re also to blame. I’m just thinking out loud on my thought process why this benchmarking has become such an old chest nut that never gets fixed, and continue to be in a vicious circle. I don’t have a solution, neither a suggestion to make. But I just want to raise a step-back and big picture thinking that why this bad custom never gets fixed. And that silicon vendors and OEMs are not the only ones responsible. Benchmark companies taking advantage of their tools to monetize, and tech media quoting benchmark scores should be taken into consideration as a part of the picture.
There's a good argument for a push towards open source benchmarks here. Unlike commercial software, theres no incentive for providing early access and such, and it would make detection-thwarting custom builds easier to obtain.
Running renamed or custom built Android bechmarks in reviews (when possible) should be the standard going forward. Given the state of the internet, where "post truth" is an understatement, I don't think SoC or phone vendors will be as worried about the PR consequences of their cheating.
There is a point to what's being stated. Take Apple for example, they have the ecosystem to mandate that an application perform in a manner otherwise not provided by using an open-ecosystem. If you use Apple, you conform to these parameters or the app isn't being submitted and shown in the app store. If you're android, or some other open system, you're saying here are some parameters, feel free to do whatever you want in terms of performance, just as long as it's not malware. It sucks on both sides, but since Apple is closed, and has generally better "forced" applications....
MediaTek's response is laughable. Reviewers don't spend hours running benchmarks on various devices to determine the performance of an SOC with certain limitations removed or with application specific optimizations. They do it to provide an idea of the capabilities of each phone in an everyday use configuration (i.e. with all the thermal and power optimizations each phone employs).
Can you explain why it's laughable? If certain SOC vendors didn't put enhancements into their software, you would have awful software. Again, take an open vs. closed ecosystem. Imagine if Microsoft said no enhancements for any vendor... how well do you think Windows would perform?
Let me point out some of the enhancements mentioned in the article: "running the memory controller at the maximum frequency all the time" and "it’s easier for workloads to have the CPU cores ramp up in frequency faster and stay there for longer period of time." Those are not realistic optimizations in a mobile device, otherwise they would be active all the time, or automatically trigger based on workload. Having them trigger for benchmarks (listed by name) but not during regular use or for other demanding loads means the benchmarks provide a false indication of performance.
The analogy to Microsoft is not valid. This would be like Intel or AMD temporarily boosting CPU performance during benchmarks but not making that same performance available to other applications. Also (I might be wrong) but when Microsoft fixes bugs or makes performance improvements to resolve issues with certain applications, I'm pretty sure that code is not restricted to only trigger with that application.
That's my point @Solar. The enhancements are application by application. There is no universal applicable usage for many things like AVX, some SSE instructions etc.. Apps often use Intel compilers vs. AMD. I don't see the outrage.
This isn't about instructions or compilers. Its about changing the power and frequency characteristics of the memory and CPU to perform better in benchmarks. Imagine if Intel adjusted its PL1 or PL2 parameters to give higher performance during benchmarks, but then reduced them for other applications. Unacceptable.
Imagine if you're selling a phone on its 120FPS capabilities, but in benchmarks it's actually locked to 60FPS because the 120FPS mode reduces overall performance - perhaps it increases power use/battery drain enough that you can't maintain high performance everywhere.
Laughable? like Cristiano Amon, Alex Rogers, Steve Mollenkopf or Fabian Gonell's (all Qualcomm's) proven lies to judge Lucy Koh! And it's about far more serious subjects than specific task scheduling for benchmarking!
But you won't read it in dedicated medias!
As you won't read Antutu v6's changes before Snapdragon 820 was released. Multicore's scores was nearly neglected when SD820 was a quadcore and Kirin and Exynos were octacores. Not task scheduling : adaptation of the benchmark to a specific soc manufacturer!
As you can read everywhere about a single security issue about Mediatek on Google's security report released in march...but not often about the 48 issues about Qualcomm!
Or PC Mark forgetting to use Cortex A72 on Helio X20 (Vernee Apollo Lite) ...PC Mark forgetting to use rk3288's VPU on video tests. This soc decodes h265/4k on every videoplayers I test and is unable to read 720p on PC Mark Video test!
A simple question... As SD660 and Helio P60 have very close scores... And have very close behavior... What does it mean in your opinion? I used Oppo A3 and my sister Xiaomi Mi A2... Every day use is the same, benchmarks are close...so where is the problem?
" SD660 and Helio P60 have very close scores... And have very close behavior..." - well thats certainly not the case with p95 and something like sd855. P95 cheated score gives pcmark 2.0 score of 9048, and oneplus 7 pro gives 9892, around 9% difference. Is there really that much difference in real life usage? Its a 2.2 ghz a75 vs 2.8/2.4ghz a76. I dont think so.
Yes, you're probably right... Even if P95 bas very good triple ISP and IA that can help in tests like photo editing, difference should be higher! I'm agree.
And 7000pts on SD710 when D1000L's score is 6800? Wake up! 2 Cortex A75 vs 4 A77 Adreno 616 vs Mali G77mp9 at least twice more powerfull 2 ISP vs 5 DSP+GPU vs 6 cores APU for IA Better video support!
If SD710 marks 7000, D1000L should mark 14000!!!
As I said with my exemple about X20 and Rk3288, you can't trust in PCMark! But all the Media spit on Mediatek! They'd better think a little, see task scheduling can't lead to 75% differences (totally absurd!!!), makes some other tests and see if the problem isn't...PCMark!
I dont know if the device you mentioned with sd710 cheats or not, dont care.
So the p95 has a better isp than 855 and that helps in photo editing scores on pcmark? About trusting pcmark or not, isnt it clear that p95 is cheating here? The non cheat score gives 30% difference compared to sd855 - this makes sense given the better cpu/gpu. I would like to see what score a cheated sd855 gives. If you want to talk about other devices redmi note 8 pro gives 10k score lol.
You can't get 30% extras with task scheduler modification, it's absurd! Any developer should ask himself some questions when seeing that kind of behavior. Try to Root a device and change governor to "performance" (in this mode, device is always at max frequency), usually you will get maybe 5% extra score on modern devices. All you win is few milliseconds of full speed frequency.
The question is : does PCMark use properly P95 unless the Rom force it? What is absolutely certain is that Dimensity 1000L is badly used. PCMark's code isn't optimised on some soc...that's a fact!
I know for Redmi Note 8 Pro, I'm writing on it... There's some funny thing with an other part of PCMark...Computer Vision Test (AI) is only 10% better than Helio P60's, I'm nearly certain some part of AI hardware is unused.
"You can't get 30% extras with task scheduler modification, it's absurd" - it is possible if you increase thermal throttling value. Also pcmark doesnt usually use all cores at max clock for its tasks. This sports mode keeps max clocks and increases throttling values, so it does seem possible. If the throttling value is higher it can keep it up for the entire test, which is like 5 minutes i think. Isnt it really the rom forcing it? What else is that code for? And it is provided by mediatek to oppo and xiaomi.
Where can i see the dimensity 1000L scores? Its not mentioned in this article, unless i missed it somehow. Its not on the ulbenchmark site either. A sd855+ rog 2 gives 14k, another result of some sort of game mode i think, its also somewhatof a cheat.
Thermal throttling won't exist on a 5 minutes test like PCMark (not on any Mediatek's at all, even with it's ten cores ontinuiously at max frequencies, X20 start throttling a little around ten minutes...maybe on SD810 on this one problems can start around 2 minutes) Modify thermal throlling can change something on longer tests (some gfxbench long term performances for exemple) But not on this one. Temperatures will be around 35 maybe up to 37 at maximum... So it's not Thermal throttling.
The only way is to boost frequencies. And this is something visible because PCMark have monitoring data Or should be visible...one more think that doesn't work properly on P95 (there are many that don't work on PCMark!!!) You can launch the test on Redmi Note 8 Pro (I do, it's on the list), score is around 9800...CPU never goes over the nominal 2.05ghz, CPU is even not at full frequency all the time, sometime it's under 0.6ghz...max temperature is 36° I try on Amlogic S912, test fail and can't display results...one more So it's not overclocking.
Yes you're right, application doesn't use all cores at max clock Root a device, tweak governor to performance mode (max frequency all the time), you will get 5% extra score on most the tests you will run. I would like to see real investigations with Antutu "cheated/not cheated", geekbench "cheated/not cheated"... I doubt differences goes over 5% If problem is only on PCMark, then it's the faulty part...the app doesn't work properly the hardware untill the system force it to do. See on detailed score. The parts mostly impacted by "cheated" mode are CPU oriented...web browsing, data manipulation, writing...my hypothesis is PCMark forget to use Cortex A75 and use A55 instead...this kind of thing can lead to 70% differences! And this kind of thing already happen for exemple on Vernee Apollo's Helio X20. In your opinion, is it normal that a benchmark, made to test capabilities of a device runs on economic cores?
On one point, You're right, my bad...confusion between so called "non cheated" score on P95 and D1000L's... Lower than P95, that mean less than 9000...impossible that PC Mark work properly in this case. 14k on SD855+ aren't strange in my opinion. Based on specifications, I expect D1000L around 12k and D1000 around 15k.
Yeah maybe it doesnt throttle much in 5 minutes, i do see the word 'throttle' in that sports mode code though, and values are different from d1000l. If all cores are at max clock, the performance does decrease a bit even in 5 minutes, atleast on my 7pro sd855 it lost 6% with the cpu throttling app under 5 minutes. Is the p95 so much better that it wont lose even 1% under same conditions? Dont think so. Thats where the low throttling of sports mode helps, if it does indeed modify throttling.
What are you talking about 37 maximum? Surely its not cpu temps right? Because those are a lot higher under max clock. Or is the p95 so great that it runs at 37 under full load?
It is indeed surprising that note 8 pro doesnt use max clock but still manages a score that matches sd855. wonder what the sport mode on it even does. When i tested on sd855, the prime core is at the lowest clock of 800mhz. Other cores werent at fullspeed either. I think the high score on rog 2 is because it forces max clocks. Its even higher than some sd865 devices.
The reason i think they dont force max clocks is because pcmark tries to find a balance between performance and battery life since its supposedly doing daily use tasks as a benchmark. They should do the non cheat version pcmark for all phones
Certainly Qualcomm has done very many shady things and has abused it's monopoly position with modem tech. I'm not defending them in any way. And I'm not suggesting that Mediatek make poor chips. I'm only calling them out for defending obvious attempts to create misleading benchmark scores.
I can't speak to other media outlets, but I believe Anandtech is one of the most objective. They have never hesitated to call out anyone in the past. Others may have, or are currently, getting away with cheating, but we should always continue to call it out at every opportunity.
Attempt is obvious... I'm against this kind of behavior too.
but extrascore should be around 5% and totally invisible in real life for users For exemple, P95 is technically very close to SD710...Antutu v8's score are very close too...220.000 and 214.000. That's the kind of difference you can see. Is it important? I don't think so...people won't care.
Problem is Anandtech start talking about 75% That's important. And in my opinion this isn't the result of cheating. That's not serious to believe it!
Anandtech analysis isn't complete. 1) on applications that work properly, it's impossible to get 30 or 75% extra performances with what Mediatek does. (We're talking about winning a few milliseconds of CPU max frequency on a ten minutes long test!) 2) see if extrascore on other benchmarks is as huge to confirm 1) 3) some investigation on other soc should be done...for exemple why SD710's score is 7000pts when twice more powerfull D1000L's is only 6800pts.
According to these points, is it reasonnable to trust in PCMark like Anandtech does?
Does Mediatek cheats to prevent badly coded benchmarks?
Why PCMark's developers don't verify their code when they see D1000L's score that doesn't fit to its technical specifications?
In playstore, I wrote to PCMark years ago about some problem on Helio X20 (unused cortex A72), answer was ridiculous. It really seems that they forget to test their work on soc from Mediatek or Rockchip...and even refuse to verify when people report problems. Just test on Realtek RTD1195, it's even worst : app crashed after video editing test... Tommorow I will try on Amlogic S912. I like very much what is tested in PCMark Work 2.0, I think it's clever...but fiability isn't there for many soc. There are many exemple.
Used to trust in Anandtech too...but not in this case. They can't be perfect all the Time.
You seem very quick to jump to conclusions. I am not an expert. But with some quick googling I can see the 75% difference (in Writing 2.0) is for a specific workload that measures the time to open, edit and save text and pdf documents. Changing how quickly the memory and CPU get to maximum frequency could make a 75% difference in this type of task. That test could also be influenced by numerous other things such as the type of internal storage and how the SOC is configured to access internal storage. The phone with the most powerful CPU cores doesn't always win every real world benchmark.
Yes, as I said in my answer to iPhonebestgamephone this 75% can happen when an application run on economic cores (here Cortex A55) instead of performance cores (A75). Impact is mostly on CPU oriented tests (not only writing but webbrowsing and data manipulation) and less on others. Writing test don't need awesome memory or storage bandwidth, it's simply text...some GPU test do
I have allready seen exactly the same problem years ago on Vernee Apollo's Helio X20...PCMark forget to use Cortex A72 on it.
If the writing 2.0 test is only using A55 cores, perhaps it is Mediatek's intention to use A55 cores in that type of workload to save battery. If that is not intentional then Mediatek needs to do a better job switching between cores based on workload. That is not controlled by PCMark.
It should be verify on some other applications with same kind of tasks...if others are OK, then the problem is obviously PCMark : simple to understand.
As Anandtech based their 30 to 75% allegations on PCMark, they could be wrong. They had to try many other apps to see differences. If you had Covid-19, took paracetamol and now you're healed, that doesn't mean paracetamol is the solution against covid-19.
This isn't seriously done.
I saw the same thing on Vernee Apollo lite's Helio X20 not only on PCMark but on Chrome too (the 2 only apps I used that had problem on a total of about 150 during 18months)...no A72... I download, Chrome beta, chrome dev, chrome canary, opera, dolfin browser...absolutely all the others used Cortex A72 without any visible impact on my autonomy. I kept Chrome Dev and that's all, end of the story. Which Mozilla Kraken's results must I refer to? The bad ones on Chrome or the very good ones on Chrome Dev...answer : the very good one on Chrome dev as it was my daily browser.
I can't trust an app that give me several abnormal results on some Mediatek's and rockchip's and crash on some Realteks and Amlogic's. This app wasn't tested enough by its developer.
You seem very hung up on those 30%-75% numbers. As you know those are on very specific workloads in one benchmark and are not intended to be the whole story.
"This isn't seriously done." You've got to be kidding. The benchmarks were listed by name in firmware. The effect of Sports mode was tested. That's more than enough for an accusation. (Unlike the human body in an open system in your COVID-19 example, it's comparatively easy to establish a cause and effect relationship of firmware in an SOC).
You want to blame app developers because Mediatek doesn't activate it's large cores? I don't know enough about software development to know whether or not that is reasonable. But if that is valid it doesn't excuse Mediatek's clear attempt to target multiple benchmark apps and modify SOC behavior to increase scores.
Of course these 30-75% are the problem... No one care about 5% differences
No I'm not kidding...effects were tested on PCMark and only PCMark. We don't know the effects on all the others. That's very simple to understand
Yes I blame PCMark's developers because I can give some exemples of abnormal behaviors or crashes!
Was Mediatek forced to cheat to prevent abnormal results? Because Qualcomm's monopoly makes some benchmarks developers specificaly test (or even developed?) on (for? I have an exemple) Snapdragon.
Nobody is ever forced to cheat, there is always a choice. Even worse is that Mediatek moved any mention of the benchmarks to some unknown location, because they know they are in trouble.
You want to provide some actual evidence for Qualcomm forcing benchmark developers to optimize for their chips? Maybe their chips are just better at certain tasks.
Maybe the issue is not with PCMark but instead with how Mediatek SOC's interpret CPU workloads? Maybe there are edge-cases that reveal weakness in Mediatek's scheduler? That is equally plausible.
A lot of maybe,yes true, maybe!...but Anandtech never use maybe... What Anandtech wrote : Mediatek cheated...results are up to 75% better. 0 analysis to understand why a so high difference. People will forget the "up to".
When these kind of thing happen with Qualcomm it was... Oneplus' fault.
No problem they were good boys, they probably get their free SD875 prototype in december 2020...
Andrei this was such a nice and eye opening read, I can't believe the magnitude to which large corporates are trying to fool us into believing what they want. Also I believe this chip scam will not end here , other makers will also be caught , till then articles like these will keep us consumers vigilant.
I hope so. Volkswagen's diesel scandal resulted in many others being caught and investigated (although what Volkswagen did was obviously much, much more damaging and serious). But I fear with SOC's it is easier to hide this sort of behaviour. Others have been caught in the past and yet here we are. Until the potential financial consequences outweigh the financial benefits this behaviour will continue in many industries.
If the Streisand Effect doesn't force MTK to clean up its act, I would hope that AnandTech refuses to review any MTK devices going forward. The resultant fallout with vendors will either force MTK to be better, or crater its sales.
I feel even if AT decides against reviewing any mobile devices with MTK SOCs, it will not have much of an effect on their sales, unless it is a collective effort among all the reviews will we see some limited effect. The reason why I said limited effect is because SOCs nowadays have reach a point that where it should run fluidly in most apps/ games (low end SOCs excluded since people buying low end phones should not be expecting high performance in the first place). When I am checking out a new phone review since a few years back, I rarely look at the performance of the SOC in benchmarks. Instead my main focus will be on the screen, battery life, and to some extent, the camera performance. The reason is because I am confident that the high end SOCs will be sufficient for my needs. Even when we look at Samsung's Exynos that was reviewed recently, sure the performance isn't as good as a SD 865, but the reality is that most people will not notice the difference in performance unless they read reviews and do a side by side comparison.
Medias aren't enough against Mediatek in your opinion?
You've heard everywhere about the single Mediatek's security issue on Google's security report released in march... ...but not often about Qualcomm's 48 security issues in the same report!
It's task scheduling...as Helio P60 and SD660's benchmarks are very close and behaviors are very close. What's your conclusion about SD660's task scheduling?
As Antutu's modify CPU tests in v6 before SD820 was released and nearly neglected multicores results when SD820 was quadcore and Kirin and Exynos were Octacores... Is it normal? Did medias write something about that?
And Mediatek doesn't even lie! You should read how Cristiano Amon, Steve Mollenkopf, Alex Rogers or Fabian Gonell (all Qualcomm's) lies to judge Lucy Koh in California. Did medias write something about it? Not many!
Or Medias doesn't insist about AV1 decoding capabilities on Dimensity 1000 when SD865 is unable? They prefer talk about 600.000pts on Antutu that 99% people won't use! When you know companies like Google, Netflix, Apple or Amazon support this open source codec? Results for most customers : unused GPU power but obsolete streaming capabilities...
People are crazy and want Monopoly! Spend their time to spit on Mediatek and stay blind about all courthouses sentences against Qualcomm.
Medias should inform about what is Standard Essential Patents and Qualcomm's duty of Fair Reasonnable And Non Discriminatory behavior. And force Qualcomm to respect FRAND!
We don't speak about a couple of points in benchmarks but about how competitors can survive in an anti-competitive market (Even Intel and NVIDIA left! And it's Qualcomm's fault! Not Mediatek's)
The Assimilator? Seems that some informations aren't assimilated properly! You can't understand what is important and what isn't!
I also wonder just how much power the Helio P95 used when maxed out by the cheat? More along the lines of "what's the harm in driving the SoC as hard as possible?" It certainly reads like it'd drain the battery in record time. Andrei, if you had the chance, please add the Wh numbers for illustration - thanks!
It will be interesting to know the full power draw. However from a normal day to day usage, these cheats will not be triggered, so will not have an impact on battery life.
Please verify if on non-cheated mode, Cortex A75 run properly. I'm nearly sure that problem comes from there... As higher score differences are visible on test where CPU is mostly involved...
I think PCMark forget use Cortex A75 and use tiny A55 instead...this and only this can lead to such a difference. If app works like it should, it's impossible to have +75% when you modify task scheduler... You should do the same with Antutu, geekbench...I'm sure "cheated mode" won't give more than 5% extra score
I already saw that problem on Vernee Apollo Lite's Helio X20...only two apps were hit (on maybe 150 I used during 18 months) by this anormal behavior : PCMark and Google Chrome (Chrome dev, Chrome beta and Chrome Canary worked perfectly) I signal the problem on Play Store, PCMark answer it was my devide's fault!!!
I don't trust in PCMark, I've seen so many strange things on it...on Mediatek, Rockchip, and recently on Realtek or Amlogic
In my opinion, write such an article based on the only PCMark score and without further investigations was a mistake.
A real shame. I have been really impressed by Mediatech's offerings so far. I had hoped to acquire more then 1 in the future ( I needed more ports then the last one offered ).
Mediatek is unfair. This is the clear reason why we should choose Qualcomm Snapdragon socs they are better in terms of performance, battery life, security and because i hate over heating phones because of Mediatek "cheapshits".
"This is the clear reason why we should choose Qualcomm Snapdragon"
No, hell no. Do you really want Qualcomm to be a monopoly? Look at Snapdragon 765G, they're still using the old 2x A76 cores thanks to their domination. Compare it to 2x A77 cores on Exy 980 5G and 4x A77 cores MTK Dimensity 1000. We need more competition, not a monopoly.
Tbh, I really don't like how Qualcomm dominated smartphone SoC. OEM should use other SoC too, as long as the performance is better if not similar and worth the price.
Unfair? And you really prefer Qualcomm...condemned all over the world...
According to European courthouse's decision, Qualcomm's behavior lead to Icera's death (nvidia's modem provider) with employement destruction in UK and France...
According to Google's security report in march, there were 48 issues on Qualcomm's and a single one on Mediatek's...of course Medias talk about the single one on Mediatek's!
I'm really happy with my Redmi Note 8 Pro's battery life and about heating on it...try one instead
You had to provide sources when you sell something... Mediatek doesn't sell you anything... Mediatek sells to OEM. Mediatek has to provide sources to OEM OEM sell you smartphones OEM have to provide you sources... True, in the past as most Mediatek's devices need to be imported, OEM didn't do often the job. As devices were sold worldwide, sources are available very often.
You can find sources for Xiaomi's (Redmi 6A, 6, Note 8 Pro, Mi Play), for Oppo's (A3, Reno Z, 2Z), for Realme's, Nokia's, Sony's (since years!)...
Conclusion : Mediatek provides sources to OEM
Please tell me how many time Mediatek was condemned?
Now if you want we can talk about courthouses decisions against Qualcomm all over the world!
I know in the past mediatek tried to charge people for access to their GPL'd code. I know meditek isn't the only one, but how many custom roms on xda do you see using mediatek compared to qualcom?
You don't know, you heard... Long time ago... I heard too, No one is able to give you a single proof of such a thing. Don't you continue to heard there's no sources for Mediatek's? Despite it's very easy to find some?
On xda, on Redmi Note 8 Pro you've got : - Lineage 17.1 - Pixel Expérience - POSP - CesiumOS - Evolution X Developed by 5 or 6 Indians. They proove it's possible. On 4pda you can find GSI Now european devs have to change their mind. Maybe it will take some time. And Mediatek need some other best sellers.
Wouldn't the obvious solution be a drain mode in the benchmark? A mode that repeats the tests until the phone shuts off? Report total score, runtime, and reason for shutoff (battery or thermals). Then you can get a score/Wh, score/time, thermals etc.
I'd like just to address why these options exist at all, which is outside the scope of the article. Benchmarks are useful for the handset maker to test the SoC and eval the thermal dissipation of their design. It is useful during handset testing to run a batch of becnches targeting various power limits and DVFS settings with the objective of measuring which, say, DVFS is the optimum for a given test - optimum for what? The result could look like this: Performance - setting 1; Efficiency - setting 2; Battery - setting 3; Maybe settings 4-45 aren't particularly good for anything. Each benchmark has it's own optimum for each objective function. At the end the Venn diagram is built and DVFS/power settings are chosen for the default mode and whatever else mode.
MTK must supply these modes as part of the SDK to the handset people. I see no contradiction, the modes are needed to eval the best compromise for real world usage. Completely agreed though, they shouldn't end up in a retail phone (particularly in the eggregious way they fake user benches), or if they do get shipped in retail they should be transparent.
Well, Anandtech is not new in reporting these things and I remember one such story long ago. Here is the URL: https://www.anandtech.com/show/7384/state-of-cheat... And when all the chipsets such as Snapdragon and Exynos are into this practice, how do you deem it a cheating? Surely this blog is intended to create a misconception in the mind of smartphone users and you can guess who would have tried this against MediaTek.
Even the dragon has records of cheating the benchmarks. But I don’t consider this cheating if some smartphone can show powerful performance at the desired point. Maybe it’s the smartphone brands that could take the credit or liability.
Even the dragon has records of cheating the benchmarks. But I don’t consider this cheating if some smartphone can show powerful performance at the desired point. Maybe it’s the smartphone brands that could take the credit or liability.
" Synthetic benchmarks are not real-world applications, but are used to represent certain usages, often with a focus on the performance of specific components within the SoC. The usual metrics are time based (seconds or frames-per-second). Not utilizing the battery saving features ensures the maximum performance of the CPU’s capabilities is demonstrated."
"Similarly, when executing a benchmark intended to assess performance, the CPU should be at its maximum performance level. If power management were still enabled, the CPU would gradually climb to its max frequency. That would be analogous to placing gradually increasing speed limit restrictions along the quarter-mile track. Like the sport car s"peed test, the “pedal” of the CPU should also be floored.
Benchmarks are made to sell hardware and everyone cheats in popular ones. Looking at fps, temperature and battery drain in real games is what has always mattered. Synthetic benchmarks are only there to be cheated, they give you a clue how the chip should perform in real life, but that's about it. Too many people take Antutu or 3DMark for gospel.
Just an innocent doubt. How do we know apple doesn't cheat in benchmarks? We can't compare it to anything equivalent and open, and can't pry out any code
From what I have seen in written reviews and review videos, gfxbench and other graphics benchmarks doesn't show the real world gaming capabilities of the gpu. The sustained tests can show the throttling behaviour but the real world gaming behaviour changes drastically as different engines are used for gaming. For example in PUBG mobile, even midrange adreno gpus are much better at running the game than the higher end mali gpus in kirin and exynos chipsets which scores very close to high end adreno gpus in gfxbench. The only benchmark test that is accurate is speedtest g by garyexplains as it uses unreal engine and unity engine which is used in real world games and in that test you can see that the gpu in kirin 980 and higher exynos chipsets are far slower than the gpu in even the midrange snapdragon 730.
People/companies that lack the knowledge to write their own benchmarking applications should not do benchmarking in the first place. If you don't have complete control over the benchmarking software you can never trust the results... Sorry Anandtech/Tom's Hardware but that's how it is.
"Lacking evidence of other SoC vendors out there enabling similar mechanisms for the device vendors...".
Here what you said about Snapdragon 865 "Meanwhile the Snapdragon S20 Ultra doesn't throttle here at all, and that is absolutely not normal – this is not a chip that is somehow super-efficient or has amazing cooling. Over the years I’ve encountered a lot of such odd results with Snapdragon phones in this benchmark, but this time around I’ve had enough of the weird behavior and I do think there’s some low-level cheating going on. The phone will actually start heating up a lot more than under other workloads, up to the point that the test will actually crash.
This confirms the attitude I have held toward phone benchmarks for a long time. I couldn't care less and neither can anyone I know. I don't know a single person who has bought a phone based on its performance on benchmarks. I have never read about anyone doing that either.
Why is the High Performance Mode an issue over here and referred to as cheating? If you can switch it on while benchmarking tests, why not switch it on while gaming? I doesn't prove Mediatek is at fault here.
The story is not new. After going through the story and comments, I doubt the credibility of such articles. These practices have been more used by Snapdragon and other proverbial names. Why only MediaTek is getting questioned? Is not it possible that Anandtech is trying to portray a picture that can benefit it from MediaTek's competitors?
Not a long time, we found stories of Snapdragon using the same techniques. I am afraid of how companies using the same strategies have biased opinions from a name like Anandtech.
I have Redmi Note8 pro and very smooth to any heavy games. And now im just wondering does redmi note 8 pro have a thing in the whole cheating thing too? Do i have to be concern about my unit?
Yes, Redmi Note 8 Pro is concerned (I use European version) Nevertheless, 30 to 75% makes me laugh!
This happen on PCMark...the reason is that this application forget that G90T has Cortex A76 and use A55 instead. If you want to do your own tests, you download a "clone app" application (for exemple the one from Winterfell Applab). This will allow to test application without "cheat". PCMark troll that way all big.LITTLE and decacore I've tested : - Vernee Apollo Lite's Helio X20 - Teclast T10's mt8176 - Oppo A3's Helio P60 - Redmi Note 8 Pro's Helio G90T
Here are Note 8 Pro's Scores : PCMark : 10245pts (cheat) - 8219 (no cheat) +25% (up to +58% on test where CPU is mostly involved like like webbrowsing) Geekbench 5 (singlecore/multicore) : 488/1619 (cheat) - 483/1518 (no cheat) +1%/+6.7%
To destroy the +58% of webbrowsing on PCMark, open chrome (it's OK on G90T), run Mozilla Kraken test...it gives you around 3300ms (lower is better), run it on Little cores, you will get around 10000ms! That prooves PCMark's webbrowsing test doesn't fit with real world!
It's sad Anandtech didn't verify PCMark fiability by other tests
Around 5% extra-perfs is that you can get with that kind of cheating method and is that you will get with any application that works properly As games are mostly based on GPU and less on CPU...difference should even be lower
I already explain to PCMark of the misuse of hardware in their application in 2018...they did nothing to correct the problem.
It seems that you're happy with smoothness on your device. It's what is important!
i just want to be short in words..so this thing is not cheating. it's an performance boost, same as qualcomm Game Turbo feature, where certain 3d games or apps (benchmarks) are boosted for an optimal experience. Intel call it Turbo Boost.. So this article is really click'n'bait and very, very low written.. by undocumented people, those kind of people that 100% their teachers used to yell at them.. I i were MediaTek i would sue this author of this article and the anandtech for false statements, ofensatory affirmations and image reputation.. very very sad..
android developer + over 8 years of experience in the field..
For the love of god, if people hail antutu or hambutu test which runs for, lets say 5 mins, if i am a manufacturer (regardless which one), how can i tell the masses that every single damn handset throttles as hell if total cpus/gpus are stressed for seconds.... I have a pocophone, i slightly loosen the facrory ultra-fail safe temp thresholds which makes it throttles insanely with basic gaming... Engineers trying to detect benchmarks and alter kernels for minutes are Normal and sane... The thing is i just want to see how fast my phone ia running at max... Pure and sfimple, people love big stuff... Reality is, phones throttle as hell and turn off mny cores to try to shape perf/power/heat/battery
Only wrong practice is somehow changing synth benchmark as in "giving numbers that aren't true"
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
111 Comments
Back to Article
FreckledTrout - Wednesday, April 8, 2020 - link
Is there some way to sandbox the benches so that the vendors can't detect the package names ot then change performance settings?Stuka87 - Wednesday, April 8, 2020 - link
If the tool is open source, one could create a custom build with enough changes to prevent their cheat software from detecting it. In the case of PCMark, thats basically what the anonymised version is that was used here.NateR100 - Wednesday, April 8, 2020 - link
This isn't even necessary, it just detects it based off the package name. Doesn't sweetest any of the actual workload or tests or anythingNateR100 - Wednesday, April 8, 2020 - link
*test*tau_neutrino - Friday, April 10, 2020 - link
You can use something like App Cloner that will create a copy of the app with a different package namepsychobriggsy - Wednesday, April 8, 2020 - link
I think one way to change benchmarking phones isn't just to run the benchmark a few times, but to run it until battery is drained, and also publish the battery life. Obviously there isn't time to do that for every benchmark, but maybe a couple would suffice to show there is a cheating issue.I.e., a phone bumping performance for benchmarks will have a lower battery life than one that isn't, and this will expose it.
Andrei Frumusanu - Wednesday, April 8, 2020 - link
PCMark already is a battery test in itself so they've already shot themselves in the foot here in this instance, and we've already hammered devices which cheat GPU with thermals, either by the phones getting incredibly hot or having issues with long-term performance.Kangal - Thursday, April 9, 2020 - link
True, but pychobriggsy has a great idea.For the benchmarks that are not all-encompassing, like GeekBench, we could fully charge the phone and then run the benchmark repeatedly. Then at the end, tally up how many runs it managed. What the average score was. How much the performance declined. The standard deviation. And even add all the bechmark results up for a Total Score.
Obviously, the devices which don't overclock much, they're going to have a much higher Total Score, as they would last much longer and have more benchmarks completed. Not to mention, the standard deviation and the performance drop will be much less, showing a nicer consistent result.
Food for thought.
eldakka - Friday, April 10, 2020 - link
I would also record device temperatures and chart that during this period. In many ways the temperature of the device could be more important than battery longevity in these sorts of tests, to see if the phone becomes too hot to physically hold it in the hand while using it, which sorta defeats the point of a phone if it becomes uncomfortable to hold it in operation.Stuka87 - Wednesday, April 8, 2020 - link
Thanks for digging into things like this guys. This will most likely not be the last of these cases that you run into. MediaTek's response was certainly disappointing.NitinYadav251096 - Thursday, April 9, 2020 - link
Yeah, not the last case and not the first one too. I read that even Snapdragon 855 and Snapdragon 730G chipsets were accused of such practices. Don't consider this a cheating for real.WaltC - Wednesday, April 8, 2020 - link
Benchmarks of all kinds--certainly not just mobile and certainly not just MTK--have been used by companies to cheat in many ways. (I'll probably carry the infamous nVidia GPU cheat of 3dMark years ago to my grave...;)) The solution? Ignore the benchmarks--well, if you must, then give them a glance, but *do not buy* based on benchmarks. Buy based on the track-record of the manufacturer--buy based on your experience with a certain manufacturer's products--buy based on the features you need and the money you want to spend. Speaking of bogus benchmarks: Userbenchmark, anyone?...;)brucethemoose - Wednesday, April 8, 2020 - link
Theres nothing wrong with some benchmark skepticism, especially for something like a phone where pure performance might not be a good metric for what a customer wants.But buying based on brand reputation and anectodal evidence isn't much of an improvement. This is doubly true for more utilitarian devices like PC hardware.
Rudde - Thursday, April 9, 2020 - link
I second this. Some phones in the mid-range has a decent GPU, others don't. Benchmarks are able to bring that to light; vendors will not do it by themselves.deil - Wednesday, April 8, 2020 - link
as an owner of note 8 pro I am dissapointed. I have it for a ~year now, its good device. And I am really sad to see they resolved to cheating.... I took it because it was cheapest usb-c equipped device with fast charge and big battery. But I did look at antutu result, and it was a factor in buying it.and now I just cannot say its good, even if my experiance is good.....
137ben - Wednesday, April 8, 2020 - link
Could you figure out if they also cheat on browser-based benchmarks like WebXPRT?Andrei Frumusanu - Wednesday, April 8, 2020 - link
No, one can't check what current website you're on unless you're in a custom browser.peevee - Wednesday, April 8, 2020 - link
So that is why iPhone is SO MUCH better in Web tests while being less better :) in synthetics? ;)And you all thought it is because of some magic Javascript engine...
RSAUser - Thursday, April 9, 2020 - link
No, it's because the iPhone can afford to make a loss on their SoC and is therefore can ship with a larger CPU, plus have a Js engine that is directly compiled for the SoC while e.g. Chrome JS engine has to be general.I think ios even beats Mac in JS tests, would have to check again though.
peevee - Monday, April 13, 2020 - link
They don't sell their SOCs, so they don't "make loss" on it.And one thing Apple cannot be accused for the last 20 years at least is that they lose any money on anything. :)
peevee - Monday, April 13, 2020 - link
"Chrome JS engine has to be general."Not really. LLVM is LLVM, and in any other way Chrome does not depend on a specific CPU. Besides, it is just ARMv8.
krazyfrog - Thursday, April 9, 2020 - link
I like how you cooked up a baseless conspiracy theory all on your own at the beginning of the comment and by the end of it had already resorted to admonishing everyone for not believing in it like it's a fact. Incredible gaslighting skills right there.eastcoast_pete - Wednesday, April 8, 2020 - link
Thanks Andrei, this and similar articles are key reasons why I come here!Three questions, one suggestion, one comment
Did you or anyone here ever had a phone "fry" while running benchmarks, especially repeated loops while connected to wall power? In other words, will some of these SoC ignore the thermal shutdown, and bake themselves to death?
Lastly, does the cheating extend to misreporting the SoC temperature in any of these?
Suggestion: I know you do that already, but please make even more noise about the thermal performance of a phone, and add FliR measurements to whatever the SoC provides on internal temperatures.
Other suggestion: Always lead the graphs and data in reviews with the "warm" mode data; the current focus on maximum burst performance with a cold SoC encourages this kind of cheating.
Last, but not least, thanks for always including the performance/Wh numbers; those are the ones I look for to evaluate whether an SoC is really performant, or artificially juiced up.
eastcoast_pete - Wednesday, April 8, 2020 - link
Damn missing edit function; two suggestions, of course.Again, great article!
Andrei Frumusanu - Wednesday, April 8, 2020 - link
1) I've had several devices in the past thermally shut down due to overheating on a cheating benchmark. I haven't had any actually "fry" themselves as the shutdown prevented that - that'd be a whole can of worms beyond what we ever saw. Also no long-term damage.2) I actually don't really use reported SoC temperatures anyhow because the sensors will always differ between devices. I use an IR thermometer to measure skin temperatures and do report these when notable.
3) The GPU data already is sorted with the phones sustained performance metrics. The efficiency data is more interesting at peak as it's supposed to be an analysis of the SoC, not device performance. Regular CPU loads don't actually thermally load the SoC sufficiently to actually throttle in most scenarios.
eastcoast_pete - Wednesday, April 8, 2020 - link
Thanks Andrei! Did any of the MTK devices go into thermal shutdown when they were stressed in "sport mode"?Also, could you take and report your temperature measurements for every device you review, not just for those that get toasty? That would help us readers put a phone's thermal performance into perspective, and reward those manufacturers that don't cheat and design their phones well. Thanks!
Andrei Frumusanu - Wednesday, April 8, 2020 - link
MediaTek SoCs don't get hot enough to overheat or throttle much anyway, the D1000 is a perfectly good chip. I have nothing against their hardware designs - this piece is all just about the software aspect of things.eastcoast_pete - Wednesday, April 8, 2020 - link
I certainly agree that their new D1000 SoC looks like a real contender, and the absence of the cheat in their software for that SoC is a good sign. With that chip having 4 A77 cores as its Big core lineup, it's difficult to imagine just how hard MTK must have driven its Helio P95 SoC to surpass the D1000 in benchmark tests. On the other hand, that result apparently made you take a close look at their software, so the D1000 already brought MTK a lot of attention, even if it's not the kind they had in mind.eek2121 - Wednesday, April 8, 2020 - link
This is one of many reasons I am working on my own custom benchmark suite. Sometimes NOT being a well known benchmark has many benefits.On the one hand, I can understand their argument. Mobile operating systems are optimized for power, and the chip may be constrained in unexpected ways because of this. However, their lack of transparency makes this argument sketchy at best.
Ray Hwang - Wednesday, April 8, 2020 - link
Not trying to defend MediaTek of any ill-intention around the benchmark score boost, but the blame shoudn’t be just on OEMs or silicon vendors. Benchmark companies are trying to make a great deal of money out of it (ripping off SoC/OEM vendors) for so-called early access, optimization, etc. And why do SoC/OEM vendors get into the trap, knowing the not-so-healthy intention in the background? Because the benchmark scores are quoted and referenced by tech media such as yourself, and it resonates with the market & industry.I’m not trying to make an argument that you’re also to blame. I’m just thinking out loud on my thought process why this benchmarking has become such an old chest nut that never gets fixed, and continue to be in a vicious circle.
I don’t have a solution, neither a suggestion to make. But I just want to raise a step-back and big picture thinking that why this bad custom never gets fixed. And that silicon vendors and OEMs are not the only ones responsible. Benchmark companies taking advantage of their tools to monetize, and tech media quoting benchmark scores should be taken into consideration as a part of the picture.
brucethemoose - Wednesday, April 8, 2020 - link
There's a good argument for a push towards open source benchmarks here. Unlike commercial software, theres no incentive for providing early access and such, and it would make detection-thwarting custom builds easier to obtain.brucethemoose - Wednesday, April 8, 2020 - link
Running renamed or custom built Android bechmarks in reviews (when possible) should be the standard going forward. Given the state of the internet, where "post truth" is an understatement, I don't think SoC or phone vendors will be as worried about the PR consequences of their cheating.hehatemeXX - Wednesday, April 8, 2020 - link
There is a point to what's being stated. Take Apple for example, they have the ecosystem to mandate that an application perform in a manner otherwise not provided by using an open-ecosystem. If you use Apple, you conform to these parameters or the app isn't being submitted and shown in the app store. If you're android, or some other open system, you're saying here are some parameters, feel free to do whatever you want in terms of performance, just as long as it's not malware. It sucks on both sides, but since Apple is closed, and has generally better "forced" applications....SolarBear28 - Wednesday, April 8, 2020 - link
MediaTek's response is laughable. Reviewers don't spend hours running benchmarks on various devices to determine the performance of an SOC with certain limitations removed or with application specific optimizations. They do it to provide an idea of the capabilities of each phone in an everyday use configuration (i.e. with all the thermal and power optimizations each phone employs).hehatemeXX - Wednesday, April 8, 2020 - link
Can you explain why it's laughable? If certain SOC vendors didn't put enhancements into their software, you would have awful software. Again, take an open vs. closed ecosystem. Imagine if Microsoft said no enhancements for any vendor... how well do you think Windows would perform?SolarBear28 - Wednesday, April 8, 2020 - link
Let me point out some of the enhancements mentioned in the article: "running the memory controller at the maximum frequency all the time" and "it’s easier for workloads to have the CPU cores ramp up in frequency faster and stay there for longer period of time." Those are not realistic optimizations in a mobile device, otherwise they would be active all the time, or automatically trigger based on workload. Having them trigger for benchmarks (listed by name) but not during regular use or for other demanding loads means the benchmarks provide a false indication of performance.The analogy to Microsoft is not valid. This would be like Intel or AMD temporarily boosting CPU performance during benchmarks but not making that same performance available to other applications. Also (I might be wrong) but when Microsoft fixes bugs or makes performance improvements to resolve issues with certain applications, I'm pretty sure that code is not restricted to only trigger with that application.
hehatemeXX - Thursday, April 9, 2020 - link
That's my point @Solar. The enhancements are application by application. There is no universal applicable usage for many things like AVX, some SSE instructions etc.. Apps often use Intel compilers vs. AMD. I don't see the outrage.SolarBear28 - Friday, April 10, 2020 - link
This isn't about instructions or compilers. Its about changing the power and frequency characteristics of the memory and CPU to perform better in benchmarks. Imagine if Intel adjusted its PL1 or PL2 parameters to give higher performance during benchmarks, but then reduced them for other applications. Unacceptable.GreenReaper - Thursday, April 9, 2020 - link
Imagine if you're selling a phone on its 120FPS capabilities, but in benchmarks it's actually locked to 60FPS because the 120FPS mode reduces overall performance - perhaps it increases power use/battery drain enough that you can't maintain high performance everywhere.Plumplum - Thursday, April 9, 2020 - link
Laughable? like Cristiano Amon, Alex Rogers, Steve Mollenkopf or Fabian Gonell's (all Qualcomm's) proven lies to judge Lucy Koh!And it's about far more serious subjects than specific task scheduling for benchmarking!
But you won't read it in dedicated medias!
As you won't read Antutu v6's changes before Snapdragon 820 was released.
Multicore's scores was nearly neglected when SD820 was a quadcore and Kirin and Exynos were octacores.
Not task scheduling : adaptation of the benchmark to a specific soc manufacturer!
As you can read everywhere about a single security issue about Mediatek on Google's security report released in march...but not often about the 48 issues about Qualcomm!
Or PC Mark forgetting to use Cortex A72 on Helio X20 (Vernee Apollo Lite)
...PC Mark forgetting to use rk3288's VPU on video tests. This soc decodes h265/4k on every videoplayers I test and is unable to read 720p on PC Mark Video test!
A simple question...
As SD660 and Helio P60 have very close scores...
And have very close behavior...
What does it mean in your opinion?
I used Oppo A3 and my sister Xiaomi Mi A2...
Every day use is the same, benchmarks are close...so where is the problem?
iphonebestgamephone - Friday, April 10, 2020 - link
" SD660 and Helio P60 have very close scores...And have very close behavior..." - well thats certainly not the case with p95 and something like sd855. P95 cheated score gives pcmark 2.0 score of 9048, and oneplus 7 pro gives 9892, around 9% difference. Is there really that much difference in real life usage? Its a 2.2 ghz a75 vs 2.8/2.4ghz a76. I dont think so.
Plumplum - Friday, April 10, 2020 - link
Yes, you're probably right...Even if P95 bas very good triple ISP and IA that can help in tests like photo editing, difference should be higher! I'm agree.
And 7000pts on SD710 when D1000L's score is 6800?
Wake up!
2 Cortex A75 vs 4 A77
Adreno 616 vs Mali G77mp9 at least twice more powerfull
2 ISP vs 5
DSP+GPU vs 6 cores APU for IA
Better video support!
If SD710 marks 7000, D1000L should mark 14000!!!
As I said with my exemple about X20 and Rk3288, you can't trust in PCMark!
But all the Media spit on Mediatek!
They'd better think a little, see task scheduling can't lead to 75% differences (totally absurd!!!), makes some other tests and see if the problem isn't...PCMark!
iphonebestgamephone - Friday, April 10, 2020 - link
I dont know if the device you mentioned with sd710 cheats or not, dont care.So the p95 has a better isp than 855 and that helps in photo editing scores on pcmark? About trusting pcmark or not, isnt it clear that p95 is cheating here? The non cheat score gives 30% difference compared to sd855 - this makes sense given the better cpu/gpu. I would like to see what score a cheated sd855 gives. If you want to talk about other devices redmi note 8 pro gives 10k score lol.
Plumplum - Friday, April 10, 2020 - link
You can't get 30% extras with task scheduler modification, it's absurd!Any developer should ask himself some questions when seeing that kind of behavior.
Try to Root a device and change governor to "performance" (in this mode, device is always at max frequency), usually you will get maybe 5% extra score on modern devices.
All you win is few milliseconds of full speed frequency.
The question is : does PCMark use properly P95 unless the Rom force it?
What is absolutely certain is that Dimensity 1000L is badly used.
PCMark's code isn't optimised on some soc...that's a fact!
I know for Redmi Note 8 Pro, I'm writing on it...
There's some funny thing with an other part of PCMark...Computer Vision Test (AI) is only 10% better than Helio P60's, I'm nearly certain some part of AI hardware is unused.
iphonebestgamephone - Saturday, April 11, 2020 - link
"You can't get 30% extras with task scheduler modification, it's absurd" - it is possible if you increase thermal throttling value. Also pcmark doesnt usually use all cores at max clock for its tasks. This sports mode keeps max clocks and increases throttling values, so it does seem possible. If the throttling value is higher it can keep it up for the entire test, which is like 5 minutes i think. Isnt it really the rom forcing it? What else is that code for? And it is provided by mediatek to oppo and xiaomi.Where can i see the dimensity 1000L scores? Its not mentioned in this article, unless i missed it somehow. Its not on the ulbenchmark site either. A sd855+ rog 2 gives 14k, another result of some sort of game mode i think, its also somewhatof a cheat.
Plumplum - Saturday, April 11, 2020 - link
Thermal throttling won't exist on a 5 minutes test like PCMark (not on any Mediatek's at all, even with it's ten cores ontinuiously at max frequencies, X20 start throttling a little around ten minutes...maybe on SD810 on this one problems can start around 2 minutes)Modify thermal throlling can change something on longer tests (some gfxbench long term performances for exemple) But not on this one.
Temperatures will be around 35 maybe up to 37 at maximum...
So it's not Thermal throttling.
The only way is to boost frequencies.
And this is something visible because PCMark have monitoring data
Or should be visible...one more think that doesn't work properly on P95 (there are many that don't work on PCMark!!!)
You can launch the test on Redmi Note 8 Pro (I do, it's on the list), score is around 9800...CPU never goes over the nominal 2.05ghz, CPU is even not at full frequency all the time, sometime it's under 0.6ghz...max temperature is 36°
I try on Amlogic S912, test fail and can't display results...one more
So it's not overclocking.
Yes you're right, application doesn't use all cores at max clock
Root a device, tweak governor to performance mode (max frequency all the time), you will get 5% extra score on most the tests you will run.
I would like to see real investigations with Antutu "cheated/not cheated", geekbench "cheated/not cheated"...
I doubt differences goes over 5%
If problem is only on PCMark, then it's the faulty part...the app doesn't work properly the hardware untill the system force it to do.
See on detailed score. The parts mostly impacted by "cheated" mode are CPU oriented...web browsing, data manipulation, writing...my hypothesis is PCMark forget to use Cortex A75 and use A55 instead...this kind of thing can lead to 70% differences!
And this kind of thing already happen for exemple on Vernee Apollo's Helio X20.
In your opinion, is it normal that a benchmark, made to test capabilities of a device runs on economic cores?
On one point, You're right, my bad...confusion between so called "non cheated" score on P95 and D1000L's...
Lower than P95, that mean less than 9000...impossible that PC Mark work properly in this case.
14k on SD855+ aren't strange in my opinion. Based on specifications, I expect D1000L around 12k and D1000 around 15k.
iphonebestgamephone - Saturday, April 11, 2020 - link
Yeah maybe it doesnt throttle much in 5 minutes, i do see the word 'throttle' in that sports mode code though, and values are different from d1000l. If all cores are at max clock, the performance does decrease a bit even in 5 minutes, atleast on my 7pro sd855 it lost 6% with the cpu throttling app under 5 minutes. Is the p95 so much better that it wont lose even 1% under same conditions? Dont think so. Thats where the low throttling of sports mode helps, if it does indeed modify throttling.What are you talking about 37 maximum? Surely its not cpu temps right? Because those are a lot higher under max clock. Or is the p95 so great that it runs at 37 under full load?
It is indeed surprising that note 8 pro doesnt use max clock but still manages a score that matches sd855. wonder what the sport mode on it even does. When i tested on sd855, the prime core is at the lowest clock of 800mhz. Other cores werent at fullspeed either. I think the high score on rog 2 is because it forces max clocks. Its even higher than some sd865 devices.
The reason i think they dont force max clocks is because pcmark tries to find a balance between performance and battery life since its supposedly doing daily use tasks as a benchmark. They should do the non cheat version pcmark for all phones
SolarBear28 - Friday, April 10, 2020 - link
Certainly Qualcomm has done very many shady things and has abused it's monopoly position with modem tech. I'm not defending them in any way. And I'm not suggesting that Mediatek make poor chips. I'm only calling them out for defending obvious attempts to create misleading benchmark scores.I can't speak to other media outlets, but I believe Anandtech is one of the most objective. They have never hesitated to call out anyone in the past. Others may have, or are currently, getting away with cheating, but we should always continue to call it out at every opportunity.
Plumplum - Friday, April 10, 2020 - link
Attempt is obvious...I'm against this kind of behavior too.
but extrascore should be around 5% and totally invisible in real life for users
For exemple, P95 is technically very close to SD710...Antutu v8's score are very close too...220.000 and 214.000. That's the kind of difference you can see. Is it important? I don't think so...people won't care.
Problem is Anandtech start talking about 75%
That's important. And in my opinion this isn't the result of cheating. That's not serious to believe it!
Anandtech analysis isn't complete.
1) on applications that work properly, it's impossible to get 30 or 75% extra performances with what Mediatek does. (We're talking about winning a few milliseconds of CPU max frequency on a ten minutes long test!)
2) see if extrascore on other benchmarks is as huge to confirm 1)
3) some investigation on other soc should be done...for exemple why SD710's score is 7000pts when twice more powerfull D1000L's is only 6800pts.
According to these points, is it reasonnable to trust in PCMark like Anandtech does?
Does Mediatek cheats to prevent badly coded benchmarks?
Why PCMark's developers don't verify their code when they see D1000L's score that doesn't fit to its technical specifications?
In playstore, I wrote to PCMark years ago about some problem on Helio X20 (unused cortex A72), answer was ridiculous. It really seems that they forget to test their work on soc from Mediatek or Rockchip...and even refuse to verify when people report problems.
Just test on Realtek RTD1195, it's even worst : app crashed after video editing test...
Tommorow I will try on Amlogic S912.
I like very much what is tested in PCMark Work 2.0, I think it's clever...but fiability isn't there for many soc. There are many exemple.
Used to trust in Anandtech too...but not in this case.
They can't be perfect all the Time.
SolarBear28 - Friday, April 10, 2020 - link
You seem very quick to jump to conclusions. I am not an expert. But with some quick googling I can see the 75% difference (in Writing 2.0) is for a specific workload that measures the time to open, edit and save text and pdf documents. Changing how quickly the memory and CPU get to maximum frequency could make a 75% difference in this type of task. That test could also be influenced by numerous other things such as the type of internal storage and how the SOC is configured to access internal storage. The phone with the most powerful CPU cores doesn't always win every real world benchmark.Plumplum - Saturday, April 11, 2020 - link
Yes, as I said in my answer to iPhonebestgamephone this 75% can happen when an application run on economic cores (here Cortex A55) instead of performance cores (A75).Impact is mostly on CPU oriented tests (not only writing but webbrowsing and data manipulation) and less on others.
Writing test don't need awesome memory or storage bandwidth, it's simply text...some GPU test do
I have allready seen exactly the same problem years ago on Vernee Apollo's Helio X20...PCMark forget to use Cortex A72 on it.
SolarBear28 - Saturday, April 11, 2020 - link
If the writing 2.0 test is only using A55 cores, perhaps it is Mediatek's intention to use A55 cores in that type of workload to save battery. If that is not intentional then Mediatek needs to do a better job switching between cores based on workload. That is not controlled by PCMark.Plumplum - Sunday, April 12, 2020 - link
It should be verify on some other applications with same kind of tasks...if others are OK, then the problem is obviously PCMark : simple to understand.As Anandtech based their 30 to 75% allegations on PCMark, they could be wrong.
They had to try many other apps to see differences.
If you had Covid-19, took paracetamol and now you're healed, that doesn't mean paracetamol is the solution against covid-19.
This isn't seriously done.
I saw the same thing on Vernee Apollo lite's Helio X20 not only on PCMark but on Chrome too (the 2 only apps I used that had problem on a total of about 150 during 18months)...no A72...
I download, Chrome beta, chrome dev, chrome canary, opera, dolfin browser...absolutely all the others used Cortex A72 without any visible impact on my autonomy.
I kept Chrome Dev and that's all, end of the story.
Which Mozilla Kraken's results must I refer to? The bad ones on Chrome or the very good ones on Chrome Dev...answer : the very good one on Chrome dev as it was my daily browser.
I can't trust an app that give me several abnormal results on some Mediatek's and rockchip's and crash on some Realteks and Amlogic's.
This app wasn't tested enough by its developer.
SolarBear28 - Monday, April 13, 2020 - link
You seem very hung up on those 30%-75% numbers. As you know those are on very specific workloads in one benchmark and are not intended to be the whole story."This isn't seriously done." You've got to be kidding. The benchmarks were listed by name in firmware. The effect of Sports mode was tested. That's more than enough for an accusation. (Unlike the human body in an open system in your COVID-19 example, it's comparatively easy to establish a cause and effect relationship of firmware in an SOC).
You want to blame app developers because Mediatek doesn't activate it's large cores? I don't know enough about software development to know whether or not that is reasonable. But if that is valid it doesn't excuse Mediatek's clear attempt to target multiple benchmark apps and modify SOC behavior to increase scores.
Plumplum - Monday, April 13, 2020 - link
Of course these 30-75% are the problem...No one care about 5% differences
No I'm not kidding...effects were tested on PCMark and only PCMark. We don't know the effects on all the others. That's very simple to understand
Yes I blame PCMark's developers because I can give some exemples of abnormal behaviors or crashes!
Was Mediatek forced to cheat to prevent abnormal results? Because Qualcomm's monopoly makes some benchmarks developers specificaly test (or even developed?) on (for? I have an exemple) Snapdragon.
SolarBear28 - Monday, April 13, 2020 - link
Nobody is ever forced to cheat, there is always a choice. Even worse is that Mediatek moved any mention of the benchmarks to some unknown location, because they know they are in trouble.You want to provide some actual evidence for Qualcomm forcing benchmark developers to optimize for their chips? Maybe their chips are just better at certain tasks.
Maybe the issue is not with PCMark but instead with how Mediatek SOC's interpret CPU workloads? Maybe there are edge-cases that reveal weakness in Mediatek's scheduler? That is equally plausible.
Plumplum - Tuesday, April 14, 2020 - link
A lot of maybe,yes true, maybe!...but Anandtech never use maybe...What Anandtech wrote : Mediatek cheated...results are up to 75% better. 0 analysis to understand why a so high difference.
People will forget the "up to".
When these kind of thing happen with Qualcomm it was... Oneplus' fault.
No problem they were good boys, they probably get their free SD875 prototype in december 2020...
aayush3298 - Wednesday, April 8, 2020 - link
Andrei this was such a nice and eye opening read, I can't believe the magnitude to which large corporates are trying to fool us into believing what they want. Also I believe this chip scam will not end here , other makers will also be caught , till then articles like these will keep us consumers vigilant.SolarBear28 - Friday, April 10, 2020 - link
I hope so. Volkswagen's diesel scandal resulted in many others being caught and investigated (although what Volkswagen did was obviously much, much more damaging and serious). But I fear with SOC's it is easier to hide this sort of behaviour. Others have been caught in the past and yet here we are. Until the potential financial consequences outweigh the financial benefits this behaviour will continue in many industries.s.yu - Wednesday, April 8, 2020 - link
Well...as far as I can tell the D1000 didn't cheat in PCMark right?Andrei Frumusanu - Thursday, April 9, 2020 - link
Correct, it didn't.s.yu - Thursday, April 9, 2020 - link
Peculiar how it's just their flagship that's not cheating.The_Assimilator - Wednesday, April 8, 2020 - link
If the Streisand Effect doesn't force MTK to clean up its act, I would hope that AnandTech refuses to review any MTK devices going forward. The resultant fallout with vendors will either force MTK to be better, or crater its sales.watzupken - Wednesday, April 8, 2020 - link
I feel even if AT decides against reviewing any mobile devices with MTK SOCs, it will not have much of an effect on their sales, unless it is a collective effort among all the reviews will we see some limited effect. The reason why I said limited effect is because SOCs nowadays have reach a point that where it should run fluidly in most apps/ games (low end SOCs excluded since people buying low end phones should not be expecting high performance in the first place). When I am checking out a new phone review since a few years back, I rarely look at the performance of the SOC in benchmarks. Instead my main focus will be on the screen, battery life, and to some extent, the camera performance. The reason is because I am confident that the high end SOCs will be sufficient for my needs. Even when we look at Samsung's Exynos that was reviewed recently, sure the performance isn't as good as a SD 865, but the reality is that most people will not notice the difference in performance unless they read reviews and do a side by side comparison.Plumplum - Thursday, April 9, 2020 - link
Medias aren't enough against Mediatek in your opinion?You've heard everywhere about the single Mediatek's security issue on Google's security report released in march...
...but not often about Qualcomm's 48 security issues in the same report!
It's task scheduling...as Helio P60 and SD660's benchmarks are very close and behaviors are very close. What's your conclusion about SD660's task scheduling?
As Antutu's modify CPU tests in v6 before SD820 was released and nearly neglected multicores results when SD820 was quadcore and Kirin and Exynos were Octacores...
Is it normal? Did medias write something about that?
And Mediatek doesn't even lie!
You should read how Cristiano Amon, Steve Mollenkopf, Alex Rogers or Fabian Gonell (all Qualcomm's) lies to judge Lucy Koh in California.
Did medias write something about it? Not many!
Or Medias doesn't insist about AV1 decoding capabilities on Dimensity 1000 when SD865 is unable?
They prefer talk about 600.000pts on Antutu that 99% people won't use!
When you know companies like Google, Netflix, Apple or Amazon support this open source codec?
Results for most customers : unused GPU power but obsolete streaming capabilities...
People are crazy and want Monopoly!
Spend their time to spit on Mediatek and stay blind about all courthouses sentences against Qualcomm.
Medias should inform about what is Standard Essential Patents and Qualcomm's duty of Fair Reasonnable And Non Discriminatory behavior.
And force Qualcomm to respect FRAND!
We don't speak about a couple of points in benchmarks but about how competitors can survive in an anti-competitive market (Even Intel and NVIDIA left! And it's Qualcomm's fault! Not Mediatek's)
The Assimilator? Seems that some informations aren't assimilated properly!
You can't understand what is important and what isn't!
C'DaleRider - Thursday, April 9, 2020 - link
Pssst.....Media is a plural word, doesn't need to have the "S" added to it to make it plural. Medium is the singular version of the word, btw.Plumplum - Friday, April 10, 2020 - link
Sorry, my bad : translation mistake! Thanks :-)In my language, Media isn't a plural word...
But what about ideas?
eastcoast_pete - Wednesday, April 8, 2020 - link
I also wonder just how much power the Helio P95 used when maxed out by the cheat? More along the lines of "what's the harm in driving the SoC as hard as possible?" It certainly reads like it'd drain the battery in record time. Andrei, if you had the chance, please add the Wh numbers for illustration - thanks!watzupken - Wednesday, April 8, 2020 - link
It will be interesting to know the full power draw. However from a normal day to day usage, these cheats will not be triggered, so will not have an impact on battery life.Andrei Frumusanu - Thursday, April 9, 2020 - link
I'll address that in the actual review of the phones, I didn't want to drag Oppo's devices too much into this.Plumplum - Saturday, April 11, 2020 - link
Please verify if on non-cheated mode, Cortex A75 run properly.I'm nearly sure that problem comes from there...
As higher score differences are visible on test where CPU is mostly involved...
I think PCMark forget use Cortex A75 and use tiny A55 instead...this and only this can lead to such a difference.
If app works like it should, it's impossible to have +75% when you modify task scheduler...
You should do the same with Antutu, geekbench...I'm sure "cheated mode" won't give more than 5% extra score
I already saw that problem on Vernee Apollo Lite's Helio X20...only two apps were hit (on maybe 150 I used during 18 months) by this anormal behavior : PCMark and Google Chrome (Chrome dev, Chrome beta and Chrome Canary worked perfectly)
I signal the problem on Play Store, PCMark answer it was my devide's fault!!!
I don't trust in PCMark, I've seen so many strange things on it...on Mediatek, Rockchip, and recently on Realtek or Amlogic
In my opinion, write such an article based on the only PCMark score and without further investigations was a mistake.
ballsystemlord - Wednesday, April 8, 2020 - link
A real shame.I have been really impressed by Mediatech's offerings so far. I had hoped to acquire more then 1 in the future ( I needed more ports then the last one offered ).
errolmoako - Wednesday, April 8, 2020 - link
Mediatek is unfair. This is the clear reason why we should choose Qualcomm Snapdragon socs they are better in terms of performance, battery life, security and because i hate over heating phones because of Mediatek "cheapshits".Fritzkier - Wednesday, April 8, 2020 - link
"This is the clear reason why we should choose Qualcomm Snapdragon"No, hell no. Do you really want Qualcomm to be a monopoly? Look at Snapdragon 765G, they're still using the old 2x A76 cores thanks to their domination. Compare it to 2x A77 cores on Exy 980 5G and 4x A77 cores MTK Dimensity 1000. We need more competition, not a monopoly.
s.yu - Thursday, April 9, 2020 - link
And, the D1000 appears to be clean in this incident.Fritzkier - Friday, April 10, 2020 - link
Yeah.Tbh, I really don't like how Qualcomm dominated smartphone SoC. OEM should use other SoC too, as long as the performance is better if not similar and worth the price.
Plumplum - Thursday, April 9, 2020 - link
Unfair?And you really prefer Qualcomm...condemned all over the world...
According to European courthouse's decision, Qualcomm's behavior lead to Icera's death (nvidia's modem provider) with employement destruction in UK and France...
According to Google's security report in march, there were 48 issues on Qualcomm's and a single one on Mediatek's...of course Medias talk about the single one on Mediatek's!
I'm really happy with my Redmi Note 8 Pro's battery life and about heating on it...try one instead
IlllI - Thursday, April 9, 2020 - link
isn't mediatek that shitty company that got caught with GPL violations? Wouldn't surprise me that the same company would do thisPlumplum - Thursday, April 9, 2020 - link
You had to provide sources when you sell something...Mediatek doesn't sell you anything...
Mediatek sells to OEM. Mediatek has to provide sources to OEM
OEM sell you smartphones
OEM have to provide you sources...
True, in the past as most Mediatek's devices need to be imported, OEM didn't do often the job.
As devices were sold worldwide, sources are available very often.
You can find sources for Xiaomi's (Redmi 6A, 6, Note 8 Pro, Mi Play), for Oppo's (A3, Reno Z, 2Z), for Realme's, Nokia's, Sony's (since years!)...
Conclusion : Mediatek provides sources to OEM
Please tell me how many time Mediatek was condemned?
Now if you want we can talk about courthouses decisions against Qualcomm all over the world!
IlllI - Friday, April 10, 2020 - link
I know in the past mediatek tried to charge people for access to their GPL'd code. I know meditek isn't the only one, but how many custom roms on xda do you see using mediatek compared to qualcom?Plumplum - Saturday, April 11, 2020 - link
You don't know, you heard...Long time ago...
I heard too, No one is able to give you a single proof of such a thing.
Don't you continue to heard there's no sources for Mediatek's? Despite it's very easy to find some?
On xda, on Redmi Note 8 Pro you've got :
- Lineage 17.1
- Pixel Expérience
- POSP
- CesiumOS
- Evolution X
Developed by 5 or 6 Indians.
They proove it's possible.
On 4pda you can find GSI
Now european devs have to change their mind. Maybe it will take some time.
And Mediatek need some other best sellers.
Timmmm - Thursday, April 9, 2020 - link
> There’s the question of whom actually came firstIt's "who"! Blindly copying other people's overcorrection is worse than just using "who" everywhere. Similarly it's "a historic" not "an historic".
Great article anyway!
MartenKL - Thursday, April 9, 2020 - link
Wouldn't the obvious solution be a drain mode in the benchmark? A mode that repeats the tests until the phone shuts off? Report total score, runtime, and reason for shutoff (battery or thermals). Then you can get a score/Wh, score/time, thermals etc.dragosmp - Thursday, April 9, 2020 - link
Good piece of investigative journalismI'd like just to address why these options exist at all, which is outside the scope of the article. Benchmarks are useful for the handset maker to test the SoC and eval the thermal dissipation of their design. It is useful during handset testing to run a batch of becnches targeting various power limits and DVFS settings with the objective of measuring which, say, DVFS is the optimum for a given test - optimum for what? The result could look like this: Performance - setting 1; Efficiency - setting 2; Battery - setting 3; Maybe settings 4-45 aren't particularly good for anything. Each benchmark has it's own optimum for each objective function. At the end the Venn diagram is built and DVFS/power settings are chosen for the default mode and whatever else mode.
MTK must supply these modes as part of the SDK to the handset people. I see no contradiction, the modes are needed to eval the best compromise for real world usage. Completely agreed though, they shouldn't end up in a retail phone (particularly in the eggregious way they fake user benches), or if they do get shipped in retail they should be transparent.
Ramandeep Singh - Thursday, April 9, 2020 - link
Well, Anandtech is not new in reporting these things and I remember one such story long ago. Here is the URL: https://www.anandtech.com/show/7384/state-of-cheat...And when all the chipsets such as Snapdragon and Exynos are into this practice, how do you deem it a cheating? Surely this blog is intended to create a misconception in the mind of smartphone users and you can guess who would have tried this against MediaTek.
GreenReaper - Thursday, April 9, 2020 - link
If everyone is cheating, it's still cheating. But are they? Or just the ones without a dragon in their phone?Tushar Sharma - Friday, April 10, 2020 - link
Even the dragon has records of cheating the benchmarks. But I don’t consider this cheating if some smartphone can show powerful performance at the desired point. Maybe it’s the smartphone brands that could take the credit or liability.Tushar Sharma - Friday, April 10, 2020 - link
Even the dragon has records of cheating the benchmarks. But I don’t consider this cheating if some smartphone can show powerful performance at the desired point. Maybe it’s the smartphone brands that could take the credit or liability.DonJulio256 - Monday, April 13, 2020 - link
Speaking of the Snapdragons (October 25 2013)https://www.qualcomm.com/news/onq/2013/10/25/power...
" Synthetic benchmarks are not real-world applications, but are used to represent certain usages, often with a focus on the performance of specific components within the SoC. The usual metrics are time based (seconds or frames-per-second). Not utilizing the battery saving features ensures the maximum performance of the CPU’s capabilities is demonstrated."
"Similarly, when executing a benchmark intended to assess performance, the CPU should be at its maximum performance level. If power management were still enabled, the CPU would gradually climb to its max frequency. That would be analogous to placing gradually increasing speed limit restrictions along the quarter-mile track. Like the sport car s"peed test, the “pedal” of the CPU should also be floored.
tkSteveFOX - Thursday, April 9, 2020 - link
Benchmarks are made to sell hardware and everyone cheats in popular ones.Looking at fps, temperature and battery drain in real games is what has always mattered.
Synthetic benchmarks are only there to be cheated, they give you a clue how the chip should perform in real life, but that's about it.
Too many people take Antutu or 3DMark for gospel.
leo_sk - Thursday, April 9, 2020 - link
Just an innocent doubt. How do we know apple doesn't cheat in benchmarks? We can't compare it to anything equivalent and open, and can't pry out any codes.yu - Thursday, April 9, 2020 - link
Can't we? There's JB for almost every version of iOS.s.yu - Thursday, April 9, 2020 - link
Also this mode turns off thermal regulations, you'd expect it to show in an Apple device as Apple's big cores are notorious for a high peak power.Fritzkier - Friday, April 10, 2020 - link
That's interesting. Maybe Anandtech could try to pry into it?sareer007 - Thursday, April 9, 2020 - link
From what I have seen in written reviews and review videos, gfxbench and other graphics benchmarks doesn't show the real world gaming capabilities of the gpu. The sustained tests can show the throttling behaviour but the real world gaming behaviour changes drastically as different engines are used for gaming. For example in PUBG mobile, even midrange adreno gpus are much better at running the game than the higher end mali gpus in kirin and exynos chipsets which scores very close to high end adreno gpus in gfxbench. The only benchmark test that is accurate is speedtest g by garyexplains as it uses unreal engine and unity engine which is used in real world games and in that test you can see that the gpu in kirin 980 and higher exynos chipsets are far slower than the gpu in even the midrange snapdragon 730.s.yu - Thursday, April 9, 2020 - link
Ugh, unreal and unity are only two engines.sareer007 - Thursday, April 9, 2020 - link
Most demanding games like pubg mobile run on these engines.iphonebestgamephone - Friday, April 10, 2020 - link
I believe 3d part in antutu is unity. And certainly k980 or higher end exynos perform much better than sd730 on antutu 3d scores.bigppp - Thursday, April 9, 2020 - link
I believe the "High Performance mode" is useful during using a SDM710 phone. When I open this mode, the system speed is better than normal.chavv - Friday, April 10, 2020 - link
Are there [lans to test other phones with "anonymised " bench?Without comparison its unknown if other manufacturers don't use same tricks
JaQua - Sunday, April 12, 2020 - link
People/companies that lack the knowledge to write their own benchmarking applications should not do benchmarking in the first place. If you don't have complete control over the benchmarking software you can never trust the results... Sorry Anandtech/Tom's Hardware but that's how it is.helloworld_chip - Sunday, April 12, 2020 - link
When Mi10 (w/ 865) was just released in Feb, it's antutu score was ~560k, which is similar to what S20 (w/ 865) shows.But later on a Xiaomi kernel update immediately pushed the score above 600k. Not sure this is pushing a specific app or pushing everything.
Suraj tiwary - Sunday, April 12, 2020 - link
"Lacking evidence of other SoC vendors out there enabling similar mechanisms for the device vendors...".Here what you said about Snapdragon 865
"Meanwhile the Snapdragon S20 Ultra doesn't throttle here at all, and that is absolutely not normal – this is not a chip that is somehow super-efficient or has amazing cooling. Over the years I’ve encountered a lot of such odd results with Snapdragon phones in this benchmark, but this time around I’ve had enough of the weird behavior and I do think there’s some low-level cheating going on. The phone will actually start heating up a lot more than under other workloads, up to the point that the test will actually crash.
GomezAddams - Sunday, April 12, 2020 - link
This confirms the attitude I have held toward phone benchmarks for a long time. I couldn't care less and neither can anyone I know. I don't know a single person who has bought a phone based on its performance on benchmarks. I have never read about anyone doing that either.Ankit Gupta - Monday, April 13, 2020 - link
Why is the High Performance Mode an issue over here and referred to as cheating? If you can switch it on while benchmarking tests, why not switch it on while gaming? I doesn't prove Mediatek is at fault here.AyushNair - Monday, April 13, 2020 - link
The story is not new. After going through the story and comments, I doubt the credibility of such articles. These practices have been more used by Snapdragon and other proverbial names. Why only MediaTek is getting questioned? Is not it possible that Anandtech is trying to portray a picture that can benefit it from MediaTek's competitors?ShauryaSharma - Monday, April 13, 2020 - link
Not a long time, we found stories of Snapdragon using the same techniques. I am afraid of how companies using the same strategies have biased opinions from a name like Anandtech.SaitamaSensei27 - Tuesday, April 14, 2020 - link
I have Redmi Note8 pro and very smooth to any heavy games. And now im just wondering does redmi note 8 pro have a thing in the whole cheating thing too? Do i have to be concern about my unit?Plumplum - Thursday, April 16, 2020 - link
Yes, Redmi Note 8 Pro is concerned (I use European version)Nevertheless, 30 to 75% makes me laugh!
This happen on PCMark...the reason is that this application forget that G90T has Cortex A76 and use A55 instead. If you want to do your own tests, you download a "clone app" application (for exemple the one from Winterfell Applab). This will allow to test application without "cheat".
PCMark troll that way all big.LITTLE and decacore I've tested :
- Vernee Apollo Lite's Helio X20
- Teclast T10's mt8176
- Oppo A3's Helio P60
- Redmi Note 8 Pro's Helio G90T
Here are Note 8 Pro's Scores :
PCMark : 10245pts (cheat) - 8219 (no cheat) +25% (up to +58% on test where CPU is mostly involved like like webbrowsing)
Geekbench 5 (singlecore/multicore) : 488/1619 (cheat) - 483/1518 (no cheat) +1%/+6.7%
To destroy the +58% of webbrowsing on PCMark, open chrome (it's OK on G90T), run Mozilla Kraken test...it gives you around 3300ms (lower is better), run it on Little cores, you will get around 10000ms!
That prooves PCMark's webbrowsing test doesn't fit with real world!
It's sad Anandtech didn't verify PCMark fiability by other tests
Around 5% extra-perfs is that you can get with that kind of cheating method and is that you will get with any application that works properly
As games are mostly based on GPU and less on CPU...difference should even be lower
I already explain to PCMark of the misuse of hardware in their application in 2018...they did nothing to correct the problem.
It seems that you're happy with smoothness on your device. It's what is important!
edbe - Monday, April 27, 2020 - link
i just want to be short in words..sothis thing is not cheating.
it's an performance boost, same as qualcomm Game Turbo feature, where certain 3d games or apps (benchmarks) are boosted for an optimal experience.
Intel call it Turbo Boost..
So this article is really click'n'bait and very, very low written.. by undocumented people, those kind of people that 100% their teachers used to yell at them..
I i were MediaTek i would sue this author of this article and the anandtech for false statements, ofensatory affirmations and image reputation..
very very sad..
android developer + over 8 years of experience in the field..
Nabil - Tuesday, July 7, 2020 - link
For the love of god, if people hail antutu or hambutu test which runs for, lets say 5 mins, if i am a manufacturer (regardless which one), how can i tell the masses that every single damn handset throttles as hell if total cpus/gpus are stressed for seconds....I have a pocophone, i slightly loosen the facrory ultra-fail safe temp thresholds which makes it throttles insanely with basic gaming...
Engineers trying to detect benchmarks and alter kernels for minutes are Normal and sane...
The thing is i just want to see how fast my phone ia running at max... Pure and sfimple, people love big stuff... Reality is, phones throttle as hell and turn off mny cores to try to shape perf/power/heat/battery
Only wrong practice is somehow changing synth benchmark as in "giving numbers that aren't true"