Unless you've been living under a rock the past couple years, you've probably heard at least something about the state of AT&T's 3G HSPA coverage in the United States. The sad reality is that dead zones exist across virtually every carrier and in every major locale. Until recently however, if one of those dead zones was your place of residence or workplace, you were either stuck paying for a network you couldn't use, or left shopping for another carrier. Bad coverage at home or work - where customers can potentially spend 70% of their time - isn't just frustrating, it's experience-killing. Mix in device-carrier exclusivity, and you can see how frustration can mount rapidly.

 

Instead of being forced to switch, carriers are hoping that users will improve coverage at their homes and small offices with femtocells. Virtually all the major carriers are betting heavily on femtocells to at least partly solve network woes - and at the same time save them billions of dollars on network buildout costs. It's a controversial move that's win-win for the carriers - users improve coverage where it matters to them on their own dime, and keep paying carriers every month for their electronic obsession. In short, they're betting that femtocells can both solve challenging coverage issues indoors and simultaneously reduce churn. 

 

Verizon and Sprint already have finished rolling out their own femtocell offerings, and AT&T is joining the fray with a nationwide deployment of their own starting mid April and lasting several months. Although mid April is when the nationwide rollout begins, there are a number of trial markets where the AT&T MicroCell is already deployed, including Arizona, where it launched Sunday March 21st. I rushed to the store the following Monday, and have been testing, hammering, and picking it apart ever since. AT&T's femtocell is far from perfect, but if your only other option is no coverage at all, it'll save you a lot of frustration.

 

Network Recap

 

Let's briefly go over the network topology itself and understand where AT&T's "Microcell" fits in:

 

 

Right off the bat, we can see that AT&T's "MicroCell" branding is actually a misnomer - it's really a femtocell. If we're being really anal about our SI prefixes, "micro" could lead you to believe that the device sits somewhere between macrocells (carrier-installed "Node B" UMTS base transceiver stations) and picocells (smaller commercial repeaters). It's an important distinction if we're to really understand where this device really fits in relation to other cellular network hardware. 

 

For some time now, AT&T has been quietly installing picocells in Apple stores across the country - they're Nokia branded boxes about 3 feet tall, a foot wide and a foot deep mounted out of sight for improving coverage where it matters. I'm told that a number of Apple store employees have affectionately nicknamed these "cancer boxes." If you look in the illustration above, that description almost matches the picocell shown in the bottom right of the center frame. It's important to note that AT&T's commercial MicroCell product isn't this. Building on thinkfemtocell's table here, I've put together a rough comparison:

 

Property	Macrocell	Picocell	Femtocell
Installation	Carrier	Carrier	Customer
Backhaul	Carrier	Carrier	Customer
Frequency Planning	Carrier	Carrier	At activation
Site Planning	Carrier	Carrier	Customer
Range	Several Blocks - Kilometers	Malls, Stores, Businesses - 10k feet or more	Homes, Small Offices - 5k feet or less
Devices Allowed	All Carrier Approved	All Carrier Approved	Customer Approved

 

While Node B antennas and picocells come with considerable setup overhead for the carriers, femtocells are entirely the consumer's responsibility - partly why the carriers love them. There's significantly less configuration that the carrier has to do; almost everything really critical happens during the activation process at power up. But we'll get into that later. 

If you're already familiar with the femtocell offerings from Sprint and Verizon, you'll find the AT&T MicroCell is much the same. It's the same premise - calls and data from phones you specify are routed over your own internet connection. Except for one small distinction - AT&T's offers 3G HSPA/UMTS data up to 3.6 Mbits/s alongside voice, where the Sprint Airave and Verizon Network Extender offer 2.5G 1xRTT CDMA2000 data at 144 Kbits/s alongside voice. Of course, that means for the AT&T MicroCell to be useful, you'll need a 3G phone; the older GSM/EDGE only iPhone 2G won't see any benefit from AT&T's femtocell. 

 

It's interesting to note that virtually all of the major carriers in the USA now offer femtocells or similar means of expanding coverage. T-Mobile is the notable exception, which foregos a femtocell in favor of Unlicensed Mobile Access (UMA) - a 3GPP standard that allows the same cellular data to be sent over any IP network, most commonly over WiFi. Let's compare the offerings from all the major providers:

 

Carrier	AT&T	Verizon	Sprint	T-Mobile
Solution	Femtocell - "3G MicroCell"	Femtocell - "Network Extender"	Femtocell - "Airave"	UMA - "HotSpot@Home"
Branding	Cisco	Samsung	Samsung	NA
Technology	3G UMTS/HSPA for voice and data	2.5G CDMA 2000 1xRTT	2.5G CDMA 2000 1xRTT	UMA voice over WiFi
Simultaneous Calls	4 Simultaneous	3 Simultaneous	3 Simultaneous	NA
Standby Approved Callers	10	100	50	NA
Data Bitrate	3.6 megabits/s (HSDPA 3.6)	144 kilobits/s	144 kilobits/s	NA
GPS Fix Required	Yes	Yes	Yes	NA
Upfront Cost	$150.00 or $50 with $100 rebate and $20/month unlimited calling plan	$249.99	$99.99	Wireless AP Cost
Hand-On/Hand-Off	No/Yes	No/Yes	No/Yes	Inter AP Handover/Yes
Coverage	5000 square feet	5000 square feet	5000 square feet	WiFi AP range
Add Ons	$20/month unlimited calling $10/month with AT&T DSL $0 with AT&T landline	None	$5/month required  $10/month unlimited calling - 1 line $20/month unlimited calling - multi line	$10/month unlimited calling

 

While Sprint and Verizon are offering virtually the same Samsung-branded product, AT&T's MicroCell is a new femtocell bearing dominant Cisco branding. The same caveats apply here: the device needs to be able to get GPS fix, meaning you'll likely have to install it near a window or in the corner of your house. Also, the hardware supports handovers from the femtocell back to the main cellular network, but calls initiated outside of femtocell coverage can never migrate or hand-on to the femtocell. Range is advertised as being 5000 square feet, and the hardware is portable; you can take it on trips or to different places so long as you register the location online. You can also sell the device to someone else - it isn't forever locked to one AT&T account. AT&T stipulates that a 1.5 Mbps downstream, 256 Kbps upstream internet connection is required.

 

Inside is just about everything you'd expect. There's a standard quick getting started guide if you're reading-challenged, an equally straightforward but more hefty users manual, power supply, and a yellow ethernet cable. Then there's the device itself, bearing Cisco branding and AT&T logos. On its front are five status LEDs: power, ethernet, GPS, computer, and 3G cellular. 

 

 

The "computer" status LED is interesting - Cisco suggests connecting the microcell in-line with your internet connection, before your router in what they call a "priority mode" configuration. This makes sense, as it offers an easy connection method for users that either don't know about QoS, NAT, or port forwarding, or aren't competent enough to configure them. I didn't investigate priority mode in much detail, instead using QoS and port forwarding rules of my own, but more on that later.

 

 

On the back are two ethernet ports. The yellow one is what we're interested in, as this is the primary connection for internet. The black one is labeled "computer," and as noted earlier and is for connecting another device behind and in line with the microcell - ostensibly a router or your computer if you lack one. There's also an antenna jack for connecting an external GPS antenna if you can't install the device near a window, or if your windows have a coating which attenuates GPS frequencies. AT&T offers absolutely no guidance of any kind about what type of port this is, but from experience it appears to be MMCX or possibly RP-MMCX. There's also a reset pin hole and power adapter. 

 

 

Interestingly, there's also an open source license agreement notice CD and sleeve. I half expected a complete dump of all the relevant code given the CD, but instead of Cisco effectively using the 700 MB of storage, they parked one 591 kilobyte, 281 page long PDF document with all the FOSS licenses for packages used on the MicroCell. No, I'm not even joking, this CD literally has one half megabyte file on it. What's even more hilarious is that the file is actually linked for download during the registration process, which makes a heck of a lot more sense. 

 

 

Of course, for your amusement, I went through the whole document and extracted all the open source packages cited. There are 24 licensed pieces of software, but the takeaway is that the femtocell is running BusyBox 1.8.2. Ostensibly, most of the code is being used for routing packets through what amounts to a router when you've put the device in "priority mode," but as we'll discover later there's plenty more going on too. 

 

In the spirit of really understanding how the AT&T MicroCell works, I was determined to get inside its inviting white shell. Unfortunately, after doing my homework, I started to get a feel for just how locked down this thing is - and why that's the case. First off, there's no internal status webpage as a diagnostic aide like you'd expect from a cable or DSL modem. Nothing. I searched around comprehensively for anything of the sort; it isn't there. What's surprising is that briefly, at startup, I saw nmap report ports 23, 80, and 8080 as filtered instead of open or closed, but that doesn't do anyone any good. The device always reports a hostname of "AT&T" and always pulls a DHCP lease at startup. There's no network configuration to speak of, so if you want to configure a static IP, static DHCP assignment is your only route. 

 

Obviously, tech savvy users also are going to want to configure proper port forwarding and QoS rules for prioritizing MicroCell traffic. Unfortunately, documentation here is beyond spartan. There are (no joke) four versions of the users guide floating around. First is the printed copy in box, then there's an AT&T PDF, and finally one in the FCC filing - all of which lack the section on what ports should be forwarded. Curiously, there's another version online that I later found here with the relevant ports (on page 5), but this was after I had already discovered them on my own.

 

Before I stumbled across that real users guide, I was determined to find out how the MicroCell was talking with AT&T and over what ports. I grabbed a second NIC and set myself up in a machine-in-the-middle configuration and started sniffing packets. It's obvious immediately that this thing is locked down tight. After booting, the device grabs a DHCP lease, syncs network time over NTP with 12.230.208.48, and does a DNS query for dpewe.wireless.att.com. After it gets the results, it talks with that server over HTTPS (TLSv1) for a bit, and then immediately fires up an IPsec VPN with 12.230.209.193. After that, there's very little we can see going on - everything happens across that VPN tunnel. 

 

 

The MicroCell uses IPsec with NAT traversal, explaining partly why you don't really have to port forward, but it's still a good idea. In fact, it's during the HTTPS session certificate exchange that we see the only bit of network traffic which would lead us to believe this is a micro, er, femtocell:

 

 

The PC302 is the core of the MicroCell, incorporating a 400 MHz ARM11 processor as well as hardware accelerator support for IPsec, ethernet, and everything you need for a femtocell. Unsurprisingly, picoChip calls this a WCDMA Femto Access Point (FAP) supporting up to 4 users, and 21 Mbps HSDPA, 5.7 Mbps HSUPA. Keep in mind here that although the controller supports a physical layer data rate at those speeds, the air interface, protocol, and implementation limit the speeds further. We'll see that in practice, speeds are lower.

It's obvious at this point that AT&T and Cisco are serious about this not being a hacker-friendly device. There's virtually no configuration status pages, little in the way of documentation about how the hardware talks to the AT&T network, and even less about what's going on under the hood. There's evidence of that hardware tamper switch too, which is surprising given the four or five blatantly open pin headers on the motherboard. 

 

Of course, the there are a number of reasons. Probably first is that AT&T doesn't want you using the MicroCell in regions they aren't licensed to operate for regulatory reasons. That's partly the reason for the GPS alongside E911 verification, though there's a less insidious reason for including a it - GPS is critical for getting very precise and accurate timing signals for the radio without expensive clock hardware. The other more serious reason for both physical and network security is that there's the very real worry that end users could gain access into the packet switched core network. The femtocell is, for all intents and purposes, a Node B base station of its own, talking with a radio network controller over lub the same way bona-fide carrier towers do. The last thing any of the carriers want is this interface being cracked or reverse engineered - all kinds of bad stuff lies that way.

As I mentioned earlier, there's a lot that takes place during device provisioning. Before anything is plugged in, the MicroCell has to be registered online with AT&T for both E911 and spectrum regulatory reasons. As I mentioned earlier, AT&T doesn't want to try and run the femtocell on spectrum they aren't licensed for, and there isn't uniform UMTS 850/1900 MHz licensing across the US. In my market, for example, AT&T is only licensed for the 1900 MHz spectrum. 

 

 

The registration begins by prompting you to enter the device's serial number. Easy enough. Next, it asks for the physical address where the device is going to be used. I was asked for that information while I was at the store, but had to enter it again during registration. This is important because it's used for E911 as well as for verification against the GPS fix the hardware gets. Finally, you enter the approved device phone numbers that will be allowed to use the microcell. At last, you're told it's ok to plug everything in, and it's time to wait - and believe me, wait you will. This is a process that can take up to an hour, though in practice at the two locations I tested, I saw times of between 20 minutes and a 45 minutes. 

 

There's a lot going on behind the scenes during this process, which explains why it can take some time. I noticed on my hardware during the first activation process that the device restarted cold at least once - it's highly likely there's a firmware update with security fixes and updated software that happens. It's also listening for and building a list of macrocells in range for its handoff list, adjusting its own transmission power, and collecting information about what frequencies are active in proximity. All the while trying to get a good GPS fix and provision itself on the network. After the GPS indicator is solid, the 3G indicators will flash either long pulses or very short pulses until the process is complete. Again, there's zero documentation about what long pulses versus short pulses means, but I saw alternating random patterns of both each time. The whole time, aside from the blinkenlights on the hardware, your only status page is the MicroCell management page in the screenshot above. There's very little in the way of communicating what's going on, it'll just let you know that activation is still pending, it's failed, or it's complete. 

 

 

Both times I did this, at the very end of the process the status page briefly communicated that the worst had happened - activation had failed. It's confusing because you'll see the 3G light go solid and your phone join the microcell, but the page will show failed for a minute. Don't panic, look down at your phone. If everything went fine, you should now see "AT&T M-Cell" if you have an iPhone or "AT&T MicroCell" on other hardware as the carrier string:

 

So obviously using the MicroCell where things are already optimal doesn't help you, in fact, it can hurt performance. On the other hand, the improvement is dramatic if you're using the MicroCell somewhere where there's no signal at all:

 

 

Performance doesn't scale linearly, and again in location 2 I wasn't in the room where the MicroCell was, I was a few rooms away. The biggest thing I can take away is that if you've got internet at your home or small office where you're installing the MicroCell, you've probably got WiFi too. At that rate, just use WiFi for data, and only tax the MicroCell with voice and SMS.

Throughout the coverage radius, voice calls were basically as good as you'd expect them to be. Of course, right at the edge of the coverage radius there was noticable distortion and blocking like you'd normally expect. That said, calling works perfectly. I started with one device, and gradually added until I got to four devices.

 

Three calls going

 

With a little math, we can really get a feel for how much bandwidth each call session is using:

 

I also initiated call between two devices on the MicroCell and let it sit to see whether the call would drop after a long time - a number of people on the AT&T forums noted issues with longer calls. I successfully got up past a half hour, after which I terminated the call. I'm confident that it would have lasted should I have left it going. 

 

I mentioned earlier that voice and data both work fine, as long as you're inside the MicroCell's coverage radius. So what happens if you step outside? In theory and per advertising, your call is supposed to transition "seamlessly" to the nearest macrosite. In fact, the first time I tried walking out of location 1, it transitioned perfectly! I then proceeded to spend over a half hour trying to repeat my initial success...

But before I dive into the problems I experienced, let's briefly touch on some of the tech behind it. There are two types of handover that happen on a cellular network: soft and hard handovers. In a soft handover, the phone is continually talking to multiple macrocells at the same time - the transition happens seamlessly because you're already connected and communicating with towers you're going to handover to. This is a make-before-break transition. The transition is simply a matter of choosing which one has the best link quality, and this is generally how most transitions happen because it's robust.

 

The other case is a hard handover - this is less seamless break-before-make transition that's nearly instantaneous in practice, but still perceptible if you know what to look for. In a hard handover, the phone literally releases its active connection and transitions to another. Also of relevance are so-called vertical handoffs - these are transitions that happen across network technologies, in this case from UMTS to GSM, though other cross-technology handovers are possible as well. There's a huge amount of complexity to these systems; effectively handing off users while maintaining voice and data sessions is a challenging multidimensional problem. One that's often taken entirely for granted because of just how well we're used to it working. 

 

So you can imagine my surprise when I discovered that the handover from femtocell back to public 3G wasn't just rough, it's downright crude. First of all, virtually every femtocell handover is of the hard variety, and in practice almost all of the ones I saw were also vertical, from UMTS (3G) to GSM (2G). During that long provisioning process, I mentioned that the femtocell is building a rough list of all the macrocells it can see. Hopefully, it can see a few of both 3G and 2G variety, which it passes onto the phone in the form of a neighbor list. The phone also builds out its own neighbor list, which you can see some of in the Field Trial (*3001#12345#*) dialer application. It's been gimped significantly since the iPhone 2G, but there's still enough here to get the point across:

 

Neighbor Cells - Just like advertised

 

Anyhow, when you leave the femtocell's radius, if there's a macrocell in range and your phone can hard handover to it, the call continues. Hopefully this is what happens, otherwise the call drops. Of course, the big caveat here is that if you're installing the femtocell in a place with little to no signal (why it's there in the first place) handover is going to be a difficult prospect. That's why I tried the AT&T MicroCell both in an area with good existing coverage, and one with poor coverage. The results at both were equally disappointing.

 

AT&T advertises an effective range of 5000 square feet. This seems about right, but it's signal dependent. Location one is just over of 2000 square feet, and continual femtocell coverage barely makes it across two rooms. Location two is effectively 5000 square feet, and femtocell coverage includes the whole house, guest house, and most of both long driveways. Again, this is largely in part because the femtocell is sitting atop existing 3G spectrum - there's no spectrum overlay just for microcells yet. The result is that if signal is bad where you are, your microcell range is larger. Conversely, if signal is already reasonable, you're not going to see more than a room or two of microcell coverage. It's a complex problem akin to the cell breathing phenomenon, and I think AT&T's 3G MicroCell product is doing a lot of ranging at setup.

So back to the AT&T MicroCell - handovers just didn't work for me. I didn't keep perfect track, partly out of frustration, partly because doing all this walking in and out and in and out of the house was physically exhausting, but to say 20% of calls handed over would be optimistic. 

 

For this test, I called the local Automated Surface Observing System (ASOS) lines for several airports and simply walked a common path outside. The ASOS stations are for pilots to get the current weather conditions at airports quickly, but I use them as simple repeatable test calls. They terminate automatically after a few minutes to prevent misuse. 

 

The most common mode of failure I saw was what's in the following video. The call starts fine, continues until the edge of microcell coverage, noticeably breaks up, and simply drops. All four of the iPhones I tested behaved the same way. Please forgive how shaky these are, it's challenging to walk rapidly, hold one phone, and take video with the other. 

 

If you're stuck in a location with absolutely no coverage, the 3G MicroCell is undoubtedly going to improve coverage, and to that extent, it does what it's supposed to do. On the flip side, if you're somewhere with relatively consistent coverage, the MicroCell is only going to frustrate you with its inconsistent at best call handover performance. As far as data is concerned, unless you have a compelling reason to, it makes more sense to use WiFi instead of 3G for both performance and battery reasons. It's definitely a feature to build a real 3G stack on the femtocell, especially when the competitor offerings are essentially voice-only 2.5G offerings, but performance is still much greater over WiFi than HSPA is, yet.

 

It's obvious that there are still a number of lingering problems with the MicroCell. The MicroCell's relatively abysmal handover performance is something which absolutely must be addressed before nationwide rollouts start in mid April, hopefully even before then. It's a challenging engineering problem, but customers are going to expect that installing what amounts to a cell tower in their home improves performance and coverage no matter what the case - handovers need to be just as transparent as they are elsewhere. There's no excuse for anything less. While we haven't tested Samsung's CDMA solution for Sprint and Verizon or T-Mobile's UMA, it's a fair bet that unreliable handovers are just as frustrating of an occurrence.

 

In the broader scope of things, there's the very real expectation that femtocells are the solution to our looming spectrum crisis. Unfortunately, this current iteration of devices doesn't let you transparently build out the public cellular network in a manner that benefits everyone - in fact, a lot of people think femtocells do just that. In the case of the AT&T MicroCell, you benefit a maximum of 10 possible people, four at a time. Verizon and Sprint offer 100 and 50 respectively, and a maximum of three at a time. Numbers like these aren't going to come close to mitigating load in even dense urban environments, because you can't share your femtocell. In a lot of ways, T-Mobile's decision to go with UMA - which is constrained solely by WiFi capacity - makes a lot of sense. In fact, it's probably a better power savings to just run one radio on the handset instead of two, and WiFi handovers from AP to AP are largely a solved problem with enterprise gear.

 

Subscribers that simply want to improve coverage for everyone, perhaps in return for being able to roam on other femtocells while moving around, this current generation of devices isn't going to satisfy. Femtocells are still in their infancy, hopefully in time we'll see apprehension about letting other phones use your bandwidth (in turn improving the network for everyone) gradually erode away. Until then, if signal is absolutely abysmal in your home or office, they're a very practical solution.