impressive article and work. I wonder if you can confirm on the iPhone x or pixel 2 that tensor processor was being used or not? Additionally, an educational article it would be very interesting if you could explain how you are doing these performance analyses on different phones!
One thing to keep in mind is precision of inference tasks. If one phone uses FP32, and another INT8, then the 4x speed difference can be explained by that alone.
Mr. Frumusanu, you are doing smt. extraordinary in today's tech media. This level of performance measuring and detailed discussion of design is largely non-existent. I can not say how much your work is appreciated.
Also, I would be very much like an article on cross platform benchmarking. Ie, Intel and Apple processor comparison show AX SoC in a very good light, however, now and then the conclusions are criticized as non valid (e.g. due to differences in architecture, memory design, os, benchmark software etc). I would like to read comprehensive discussion on this that would sort things out at least a bit.
"Performant" has been recognized in dictionaries for decades. It's in Oxford, Webster's, Cambridge, and Collins. Only American Heritage hasn't recognized it yet, but they pride themselves on taking 20+ years after a word is in common usage.
'-ant' already exists, and is a real suffix. 'Perform' already exists, and is a real word. You can combine word-parts in many different ways, enough that a dictionary would require an order of magnitude more pages if we wanted to collect each possibility.
Well. The GPU can consume up to 6W, single thread CPU load can go up to 4W and NPU consumes up to 5,5 W.
And TDP is like 3-4W ? This cannot continue forever like that, transactional load optimized or not. Then how much of all those different parts of the chip can be used at once without blowing up ? How well would that fit into a laptop ? What if a three year torn up and disolved battery has to power this ?
Considering A12 actually matches A10X quite easilly, I think A12X is going to be truly desktop grade.
I find the expected switch of the new iPad Pros from Lightning to USB-C somewhat peculiar, too.
That port change to the iPads is still just a rumour at this point, but it could fit perfectly into a strategy of announcing the Mac platform switch at next WWDC and offering a macOS developer beta for those iPad Pros until actual ARM Macs were available later in the year.
The iPads could then be used as interim developer Macs with a standard USB-C port so you could easily connect standard Mac peripherals (plus Bluetooth for keyboard and mouse).
It's not a strong piece of evidence, it would just fit rather effortlessly.
"That port change to the iPads is still just a rumour at this point, but it could fit perfectly into a strategy of announcing the Mac platform switch at next WWDC and offering a macOS developer beta for those iPad Pros until actual ARM Macs were available later in the year."
ARM based Mac has been rummer for years maybe even a decade. Even though the original mac's were closer to RISC machines with 68xxx and PowerPC series - but those days are long gone.
I have yet to see a real performance comparison of same benchmark of ARM vs x86 based CPU's. I am not talking about Web stuff
I just think the ARM cores are used for different purpose - IE Apps don't needs the power of desktop applications.
Plus something completely off the CPU's environments - it appears that MacOS line has not even done anything with touch screens and iOS line has not done anything with mouse and keyboard. It appears Apple is trying to keep them separated - maybe because they would love one day to kill the MacOS line.
Like any thing else you see on internet, this is just my opinion.
ARM based Mac has been rummer for years maybe even a decade.
It is almost certain that there are a few MacBooks at Apple with prototype Axx boards running experimental macOS builds, just for evaluation. Those rumours have been popping up consistently enough to make that plausible, but those will not be mass-produced like that. So the usual sequence would be an announcement at WWDC and some preliminary developer machines – back then it was PC motherboards in a PowerMac case, no it could be regular iPads conscripted for the task.
I have yet to see a real performance comparison of same benchmark of ARM vs x86 based CPU's. I am not talking about Web stuff
Geekbench with its suite of native tests puts the A12 cores at about desktop i5 level. It's mostly passive cooling and tiny pocket-sized batteries limiting multicore performance.
I just think the ARM cores are used for different purpose - IE Apps don't needs the power of desktop applications.
That's long in the past. Today they do, especially on iOS.
Plus something completely off the CPU's environments - it appears that MacOS line has not even done anything with touch screens and iOS line has not done anything with mouse and keyboard. It appears Apple is trying to keep them separated - maybe because they would love one day to kill the MacOS line.
That wouldn't change in my speculative scenario – iPads as developer Macs would not use their touch capability but would be used with keyboard and mouse/trackpad.
And especially looking at Windows I still think the separation keeps making sense.
Like any thing else you see on internet, this is just my opinion.
Of course, and the same applies to my own posts! 🙂
" I think A12X is going to be truly desktop grade."
Maybe for some applications, but real desktop applications and not just apps - ARM basically does not have the power to replace it.
A good sign of this when iPad Pro can actually create applications for the iPad Pro and iPhone. I don't believe development tools are available that actual run on iPad Pro. I am talking about native development and not across a VPN or other remote connection
Maybe for some applications, but real desktop applications and not just apps - ARM basically does not have the power to replace it.
The Axx CPUs don't use licensed ARM cores, they're just compatible to the ARMv8 instruction set. And that is not a limitation – quite the contrary!
A good sign of this when iPad Pro can actually create applications for the iPad Pro and iPhone. I don't believe development tools are available that actual run on iPad Pro. I am talking about native development and not across a VPN or other remote connection
That is not limited by CPU performance but by touch UI paradigms and the current state of code signing requirements under iOS.
Even the soon to be replaced current iPads are already more powerful than regular developer machines only a few years back.
"I’ve gathered some incomplete SPEC numbers on Arm’s A55 (it takes ages!) and in general the performance difference here is 2-3x depending on the benchmark"
Nice, but how much more power the tempest core consume vs those -very slow- A55 (or even A53)?
But surprisingly not THAT efficient. Half the energy for a quarter of the performance? That's a much worse tradeoff than I was expecting. I'm guessing this means one of - either Apple tunes the OS so that the small core USUALLY runs at a lower (and much more efficient) frequency? OR - a SPEC2006 type of workload exercises the uncore in a way that Apple does not expect for its uses of the small cores, and much of the energy is being burned in that uncore?
More generally, DAMN Andrei! I think we are all incredibly grateful that you joined Ars. I hope you'll stay on the Apple beat for many years! And that you'll expand your domain to cover similar analyses of the new iPads (and, what the hell, the new Apple Watch; why not even the Apple TV!)
I recon the little cores will rarely be used at near load power consistently, as those tasks would more likely move to a big core. At the lower ranges of operations, they're probably far more efficient.
I've been thinking about this more, and that energy usage really seems high! I think when the new iPads come out, an interesting variant might be, instead of running SPEC2K6 on the small cores, rather running SPECRate2006 on the entire machine.
For a phone we'd expect (vigorous handwaving) ~3x the single threaded performance, (2x big core, 4x quarter core) slightly lower bcs of reduced frequency; but we'd also get composite power and energy numbers. (Presumably the iPad will have a different large and small core config, but you get the idea.)
If those power and energy numbers are very different from the single threaded numbers here, we'd at least know that there's something missed in the methodology.
Thank you Andrei Frumusanu for this detailed article. I enjoyed your detailed work on the A12 in the Iphone article as much as this. Keep it up and we are many who really love these kinds of deep dives.
Will there be any additional testing with regards to XMM Modem and Antenna issues? Would love a few words on this and I am waiting to see if Xr 2x2 Antenna with Aluminium has better reception than Xs.
Thanks Andrei, really appreciate this addition to the detailed review of the XS and XS Max, especially of the SoC! Question: While the high AI scores of the A12's large NPU are sort-of expected, I was surprised by how well the 845 did (of cause, still dwarfed by the A12). Does it contain neural processing-type elements, or is the 845 using its Hexagon DSP and/or GPU to execute the AI benchmark tests?
QC introduced an NPU with the Snapdragon 820, At the time this was branded Zeroth, which was supposed to be a generic brand name for QC AI IP but for the usual incomprehensible QC reasons, that branding, and the attempt to tell the world that QC has its own NPU, like it has its own GPU, disappeared a few weeks later without a trace.
Thanks! Yes,QC seems to be uniquely incapable in blowing its own horn of its flagship SoC's AI/neural processing abilities. I wonder how many (if any) phone manufacturers are utilizing the neural processing functions of the snapdragon chips in their phones, and, if yesm to what extent.
What do you make of the link I attached in my comment? Is your point that running it how QC does (across some combination of CPU, DSP and GPU) doesn't count as "real" NPU? That's true is HW terms, but does it matter in consequence terms?
I certainly don't care either way, but I'd prefer to see benchmarks before getting upset. This space has far too much complaining about the words that people use ("real" RISC vs "real" CISC) rather than ignoring the words and looking at the outcomes, regardless of how they are achieved.
Excellent work Andrei! I guess you activated the low power mode to benchmark the small cores. iPhone users posted GB4 results where the iPhone reached about 70% of the normal values in LPM. Your findings for SpecInt2006 in LPM are about 25%. This does not fit together at all...
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
Facebook
Twitter
RSS
32 Comments
Back to Article
mmrezaie - Friday, October 12, 2018 - link
impressive article and work. I wonder if you can confirm on the iPhone x or pixel 2 that tensor processor was being used or not? Additionally, an educational article it would be very interesting if you could explain how you are doing these performance analyses on different phones!Speedfriend - Friday, October 12, 2018 - link
This is why I like to read Anandtechp1esk - Friday, October 12, 2018 - link
One thing to keep in mind is precision of inference tasks. If one phone uses FP32, and another INT8, then the 4x speed difference can be explained by that alone.martm - Friday, October 12, 2018 - link
Mr. Frumusanu, you are doing smt. extraordinary in today's tech media. This level of performance measuring and detailed discussion of design is largely non-existent. I can not say how much your work is appreciated.Also, I would be very much like an article on cross platform benchmarking. Ie, Intel and Apple processor comparison show AX SoC in a very good light, however, now and then the conclusions are criticized as non valid (e.g. due to differences in architecture, memory design, os, benchmark software etc). I would like to read comprehensive discussion on this that would sort things out at least a bit.
Constructor - Sunday, October 14, 2018 - link
I'd expect that topc to ramp up with expectations that Apple might actually pursue such a platform switch for their Macs.Elstar - Friday, October 12, 2018 - link
"performant" is a made up word. Did you mean "fast" or "efficient" or both?PS – A good way to remember this: "if a performer performs at a performance, is she performant?"
Ishwa - Friday, October 12, 2018 - link
If a performer performs at a performance most rapidly, she is the most performant performer.Death666Angel - Friday, October 12, 2018 - link
""performant" is a made up word." - ah, as opposed to all those word that naturally grow on trees?porcupineLTD - Friday, October 12, 2018 - link
Thank you! Very insightful.BrooksT - Friday, October 12, 2018 - link
"Performant" has been recognized in dictionaries for decades. It's in Oxford, Webster's, Cambridge, and Collins. Only American Heritage hasn't recognized it yet, but they pride themselves on taking 20+ years after a word is in common usage.mkozakewich - Sunday, October 14, 2018 - link
'-ant' already exists, and is a real suffix. 'Perform' already exists, and is a real word. You can combine word-parts in many different ways, enough that a dictionary would require an order of magnitude more pages if we wanted to collect each possibility.GC2:CS - Friday, October 12, 2018 - link
Well. The GPU can consume up to 6W, single thread CPU load can go up to 4W and NPU consumes up to 5,5 W.And TDP is like 3-4W ? This cannot continue forever like that, transactional load optimized or not.
Then how much of all those different parts of the chip can be used at once without blowing up ? How well would that fit into a laptop ? What if a three year torn up and disolved battery has to power this ?
Considering A12 actually matches A10X quite easilly, I think A12X is going to be truly desktop grade.
Constructor - Sunday, October 14, 2018 - link
I find the expected switch of the new iPad Pros from Lightning to USB-C somewhat peculiar, too.That port change to the iPads is still just a rumour at this point, but it could fit perfectly into a strategy of announcing the Mac platform switch at next WWDC and offering a macOS developer beta for those iPad Pros until actual ARM Macs were available later in the year.
The iPads could then be used as interim developer Macs with a standard USB-C port so you could easily connect standard Mac peripherals (plus Bluetooth for keyboard and mouse).
It's not a strong piece of evidence, it would just fit rather effortlessly.
HStewart - Monday, October 15, 2018 - link
"That port change to the iPads is still just a rumour at this point, but it could fit perfectly into a strategy of announcing the Mac platform switch at next WWDC and offering a macOS developer beta for those iPad Pros until actual ARM Macs were available later in the year."ARM based Mac has been rummer for years maybe even a decade. Even though the original mac's were closer to RISC machines with 68xxx and PowerPC series - but those days are long gone.
I have yet to see a real performance comparison of same benchmark of ARM vs x86 based CPU's. I am not talking about Web stuff
I just think the ARM cores are used for different purpose - IE Apps don't needs the power of desktop applications.
Plus something completely off the CPU's environments - it appears that MacOS line has not even done anything with touch screens and iOS line has not done anything with mouse and keyboard. It appears Apple is trying to keep them separated - maybe because they would love one day to kill the MacOS line.
Like any thing else you see on internet, this is just my opinion.
Constructor - Tuesday, October 16, 2018 - link
It is almost certain that there are a few MacBooks at Apple with prototype Axx boards running experimental macOS builds, just for evaluation. Those rumours have been popping up consistently enough to make that plausible, but those will not be mass-produced like that. So the usual sequence would be an announcement at WWDC and some preliminary developer machines – back then it was PC motherboards in a PowerMac case, no it could be regular iPads conscripted for the task.
Geekbench with its suite of native tests puts the A12 cores at about desktop i5 level. It's mostly passive cooling and tiny pocket-sized batteries limiting multicore performance.
That's long in the past. Today they do, especially on iOS.
That wouldn't change in my speculative scenario – iPads as developer Macs would not use their touch capability but would be used with keyboard and mouse/trackpad.
And especially looking at Windows I still think the separation keeps making sense.
Of course, and the same applies to my own posts! 🙂
HStewart - Monday, October 15, 2018 - link
" I think A12X is going to be truly desktop grade."Maybe for some applications, but real desktop applications and not just apps - ARM basically does not have the power to replace it.
A good sign of this when iPad Pro can actually create applications for the iPad Pro and iPhone. I don't believe development tools are available that actual run on iPad Pro. I am talking about native development and not across a VPN or other remote connection
Constructor - Tuesday, October 16, 2018 - link
The Axx CPUs don't use licensed ARM cores, they're just compatible to the ARMv8 instruction set. And that is not a limitation – quite the contrary!
That is not limited by CPU performance but by touch UI paradigms and the current state of code signing requirements under iOS.
Even the soon to be replaced current iPads are already more powerful than regular developer machines only a few years back.
dudedud - Friday, October 12, 2018 - link
"I’ve gathered some incomplete SPEC numbers on Arm’s A55 (it takes ages!) and in general the performance difference here is 2-3x depending on the benchmark"Nice, but how much more power the tempest core consume vs those -very slow- A55 (or even A53)?
Andrei Frumusanu - Friday, October 12, 2018 - link
Tempest is more efficient.name99 - Friday, October 12, 2018 - link
But surprisingly not THAT efficient. Half the energy for a quarter of the performance? That's a much worse tradeoff than I was expecting. I'm guessing this means one of- either Apple tunes the OS so that the small core USUALLY runs at a lower (and much more efficient) frequency? OR
- a SPEC2006 type of workload exercises the uncore in a way that Apple does not expect for its uses of the small cores, and much of the energy is being burned in that uncore?
More generally, DAMN Andrei! I think we are all incredibly grateful that you joined Ars.
I hope you'll stay on the Apple beat for many years! And that you'll expand your domain to cover similar analyses of the new iPads (and, what the hell, the new Apple Watch; why not even the Apple TV!)
tipoo - Sunday, October 14, 2018 - link
I recon the little cores will rarely be used at near load power consistently, as those tasks would more likely move to a big core. At the lower ranges of operations, they're probably far more efficient.name99 - Wednesday, October 17, 2018 - link
I've been thinking about this more, and that energy usage really seems high!I think when the new iPads come out, an interesting variant might be, instead of running SPEC2K6 on the small cores, rather running SPECRate2006 on the entire machine.
For a phone we'd expect (vigorous handwaving) ~3x the single threaded performance, (2x big core, 4x quarter core) slightly lower bcs of reduced frequency; but we'd also get composite power and energy numbers. (Presumably the iPad will have a different large and small core config, but you get the idea.)
If those power and energy numbers are very different from the single threaded numbers here, we'd at least know that there's something missed in the methodology.
mikael.skytter - Saturday, October 13, 2018 - link
Thank you Andrei Frumusanu for this detailed article. I enjoyed your detailed work on the A12 in the Iphone article as much as this. Keep it up and we are many who really love these kinds of deep dives.iwod - Saturday, October 13, 2018 - link
Will there be any additional testing with regards to XMM Modem and Antenna issues? Would love a few words on this and I am waiting to see if Xr 2x2 Antenna with Aluminium has better reception than Xs.eastcoast_pete - Sunday, October 14, 2018 - link
Thanks Andrei, really appreciate this addition to the detailed review of the XS and XS Max, especially of the SoC!Question: While the high AI scores of the A12's large NPU are sort-of expected, I was surprised by how well the 845 did (of cause, still dwarfed by the A12). Does it contain neural processing-type elements, or is the 845 using its Hexagon DSP and/or GPU to execute the AI benchmark tests?
name99 - Sunday, October 14, 2018 - link
QC introduced an NPU with the Snapdragon 820, At the time this was branded Zeroth, which was supposed to be a generic brand name for QC AI IP but for the usual incomprehensible QC reasons, that branding, and the attempt to tell the world that QC has its own NPU, like it has its own GPU, disappeared a few weeks later without a trace.Regardless, the NPU continued, and so, yes, the 845 likewise has an NPU, called the "QC Neural Processing Engine".
https://www.engadget.com/2017/12/06/qualcomm-snapd...
eastcoast_pete - Sunday, October 14, 2018 - link
Thanks! Yes,QC seems to be uniquely incapable in blowing its own horn of its flagship SoC's AI/neural processing abilities. I wonder how many (if any) phone manufacturers are utilizing the neural processing functions of the snapdragon chips in their phones, and, if yesm to what extent.Andrei Frumusanu - Monday, October 15, 2018 - link
No existing Snapdragon has an NPU.name99 - Monday, October 15, 2018 - link
What do you make of the link I attached in my comment?Is your point that running it how QC does (across some combination of CPU, DSP and GPU) doesn't count as "real" NPU? That's true is HW terms, but does it matter in consequence terms?
I certainly don't care either way, but I'd prefer to see benchmarks before getting upset. This space has far too much complaining about the words that people use ("real" RISC vs "real" CISC) rather than ignoring the words and looking at the outcomes, regardless of how they are achieved.
ben003 - Wednesday, October 17, 2018 - link
Excellent work Andrei! I guess you activated the low power mode to benchmark the small cores. iPhone users posted GB4 results where the iPhone reached about 70% of the normal values in LPM. Your findings for SpecInt2006 in LPM are about 25%.This does not fit together at all...
ben003 - Thursday, October 18, 2018 - link
Link for GB4 results were missing:https://wccftech.com/iphone-xs-max-lower-power-mod...
rama_31 - Friday, October 19, 2018 - link
Is iphone xs really worth it to buy for the customer who have iphone x?