Digium Confirms Major Issues with AA50 VoIP Appliance: Spotaneous Reboots and Memory Card Write-Lock, A Review

Let me begin by saying that I am a big fan of Digium in general. All of their products that I have used thus far in several IP telephony / VoIP installations have worked tremendously well.

Background: Doug and I recently integrated a Digium AA50 phone system appliance with Polycom IP telephones for a small business with moderate call volume (approximately 50-60 outgoing long-distance calls daily and many more incoming calls). The system is attached to 3 analog Embarq telephone lines (with rollover from one main number) via the AA50’s FXO ports and a 2-call limit VoIP trunk with CallCentric.

The phone network is powered, VLAN’d and QoS’d by a Dell PowerConnect 3548P that is capable of delivering Power-over-Ethernet (PoE) to all 48 ports. Users of the system can plug their laptops into the provided and automatically VLAN’d Ethernet pass-thru on both the SoundPoint IP601 and IP331 phones; this is a really nice feature of the SoundPoint models we use and allows you to only have one jack for two devices while still keeping voice and data on separate VLANs. The 3548P switch is really slick in that it came with multiple VoIP provider VLANs programmed in already (based on MAC address); any devices from those manufacturers are automatically placed into the voice VLAN on the switch. Oddly enough, the Dell switch did NOT have an entry for Polycom, though it did come with presets for Cisco, Aastra, Avaya and a handful of others.

The AA50 phone switch, Dell 3548P and associated firewall (a Linksys WRT54GL running OpenWRT for VPN, QoS and VLAN capability) and DSL modem are all backed up by an APC UPS SUA750XL that, at current load, gives the customer about 90 minutes of runtime during a power outage.

We utilize the licensed g729.a codec internally and over our VoIP trunks with CallCentric to minimize bandwidth consumption.

The AA50 appliance is used with (3) Polycom SoundPoint IP601 with sidecars for receptionists, (1) Polycom SoundPoint IP6000 conference phone and (10) Polycom SoundPoint IP331 phones. Having worked with interfacing Avaya, Aastra and Polycom phones to Asterisk in the past (trixbox, Asterisk Business Edition and the like), I will continue to pick Polycom hands-down for interoperability with Asterisk, with Aastra as my second choice. Avaya makes amazing systems, but they add so much proprietary stuff on top of SIP that I have found that the phones are more trouble to work with than Polycom or Aastra, which adhere much more closely to established standards.

Review: On the AA50, usability would have received 5 stars, except that some advanced setup functions still require getting under the hood and modifying config files by hand. One setting that should be built into the GUI is a limit for number of simultaneous calls on a given trunk; to do so requires editing users.conf by hand. I do not believe that it is possible to add more than one VoIP trunk provider via the web GUI at this time either, but perhaps I missed something.

Another thing that hurt usability is the location of stored config files on the system. Because this system uses a CF card, files are not located in the places you would expect to find them on an Asterisk system. Once you figure out that the CF card is mounted in /var/lib/asterisk/sounds/, you are good to go. It is important to note that some normal Asterisk functionality (such as CDR) does not appear to be functional because of CF card limitations.

I would have liked to have seen in the web GUI and documentation a better reference to the “advanced options” switch that allows you to do such things as modify config files using the web GUI. This option is hidden in the menu until you explicitly enable advanced options, which is annoying.

Pricing is ok on the AA50 if you are not able or willing to put together your own whitebox solution. If you know what you are doing, hindsight tells me that I would now recommend going the whitebox route; you can build two systems and have one for a spare for the retail price that the AA50 sells at.

Unfortunately, the AA50 does not support sending syslog events to a remote syslog server. SNMP is not available that I am aware of and enabling the “full logging” in logger.conf is futile; the full log gets wiped when the AA50 reboots because the log is stored in volatile RAM and not on the CF card; there is no way to redirect the log that I am aware of. Currently, I have a cron process that logs into the AA50 every few minutes and grabs the latest log information.

***ISSUES: As of the date of this writing (02 Nov 2009), we have two major issues which Digium has confirmed are issues for multiple customers: Issue 1) The system spontaneously reboots, whether or not calls are currently in progress. Fortunately, the system reboots very quickly (~20-30 seconds) and it is almost unnoticeable if you have low call volume. It takes phones a few seconds to re-register with the server though and so phones are out of service for upwards of a minute when a spontaneous reboot happens. Apparently this spontaneous reboot situation is attributed to a memory leak, but Digium is not aware of the exact problem and does not have a resolution as of yet. I am told by Digium that they previously had a memory leak issue and remedied it quickly, but are having a harder time tracking it down this time.

Issue 2) The CF memory card will spontaneously change to read-only mode while the phone system is in operation. As the “hard drive” and only permanent storage in the system, this is a huge issue because when the CF card is mounted read-only, you cannot leave or delete voicemails. The system will function as if nothing is wrong, but then fail to actually write or delete voicemail. At that time, you have to umount the CF card and remount it using the command line or reboot using the web interface. Another related symptom: If the CF card goes to read-only mode while someone is leaving a voicemail, that voicemail will be corrupted and you will not be able to delete or use the space occupied by the voicemail until you delete the file manually using the command line.

Another really quirky issue with the AA50 is that the WAN and LAN ports both share the same MAC address and there is no way that we could figure out to override this permanently. This makes things really interesting when you plug the WAN and LAN ports into the same VLAN’d switch; trust me, it is not pretty. We ended up routing all traffic via the LAN port and do not use the WAN port for anything. This required a little creativity and I would be happy to share our config if anyone is interested.

We purchased a Gold support contract with Digium because it provides advanced hardware replacement in the event of failure and it took several weeks to actually receive the support certificate for some unknown reason. Digium even told me at one point that they did not even sell support for the AA50 any more, even though they still showed this offering for sale on their own web site! Not receiving the certificate might have prevented us from getting support from Digium, but the tech I ran across was kind enough to give us a pass and address our issue even though he could not verify that I had purchased support.

Overall, support from Digium has been better than what you would receive from a typical tech company, but it has not been stellar. Techs have followed up with me while I have an open ticket, but it takes them quite a while to do so and this does not seem to be standard practice, but more just getting a good tech (i.e. it might not be policy).

When I call Digium support, I always get someone right away, but I am usually told that I will be put into a callback queue after my contact and issue details have been taken. It seems that Digium has little resources dedicated to supporting the AA50 appliance and not many Digium techs I have spoken with seem as comfortable with this unit as they are with the Switchvox product line. As a matter of fact, it often seems that I know just about as much about this unit as Digium does and that is not comforting. Typically, I receive a callback within 1 hour, but this can be an eternity if you have a terribly broken system and can not take or make calls. For the high cost of the support agreements, better service should be afforded to the customer.

I will wrap up by saying that our reseller, VoIP Supply (Arthur Miller in particular) have been stellar. Arthur was tremendously helpful during pre-sales, ushered and expedited our order after-hours when we needed equipment ASAP (and called at the 11th hour) and he even helped to escalate these issues with Digium. Arthur has done a tremendous job and I can wholeheartedly recommend VoIP Supply to others. If you would like to contact Arthur for purchases, please let me know and I will send you a private message with his contact details.

I will update with another review when we have a resolution to these issues.

13 Responses to “Digium Confirms Major Issues with AA50 VoIP Appliance: Spotaneous Reboots and Memory Card Write-Lock, A Review”

  1. Hey there Dennis,
    Great review! This is both honest and insightful. Just wanted to leave a comment about your service provider trouble with the GUI. The Asterisk GUI *should* be able to configure multiple VoIP Service Providers. I just tested my local installation and was able to successfully configure two. If yours doesn’t however, feel free to contact Tech Support and report it as a bug.
    To also note about the ‘advanced options’: It was intentionally “hidden” as we didn’t want those that aren’t advanced to either easily confuse themselves or unintentionally break their system. There was the assumption that those who are advanced would be easily able to find how to enable the advanced options. If that is not the case, then maybe we can consider a better method of presenting this option.
    I’d love to follow up with you. Feel free to visit me in IRC, I’m ‘awk_r’ in #asterisk-gui (see asterisk.org for IRC details).

  2. Hey Ryan,

    Thank you for the kind reply.

    I do now see the button to add more SIP/IAX trunks, even after I already have one trunk configured. I completely missed this previously for some reason (or maybe in an old firmware); thank you for the offer to help!

    I think that Digium’s decision to make the “advanced options” hidden by default is a good call and I can see why things were done the way that they were. I interact with the advanced options regularly and three clicks was just a minor annoyance for me. For those of you reading this who are unfamiliar with the AA50 options, keep in mind that some of the advanced options offered therein are still somewhat experimental in the community at large and as such, not supported by Digium or seen as stable (rightfully so).

    One giant gold star for Digium: The guys at Digium I have worked with thus far have been extremely supportive of the minor custom macros and changes that we made on our AA50 platform. Other lesser tech companies routinely put the blame on the customer who does anything evenly remotely out of the ordinary; I do not in any way feel that this is the case with Digium.

    While waiting on the reboot issue to be identified and fixed, a simple reboot script based on Linux expect (“apt-get install expect” on Ubuntu) seems to be keeping us stabilized and I will be following up here with my script later today.

    I look forward to catching up with you on IRC Ryan.

    -Dennis

  3. Excellent post, keep at it!

  4. Hi Dennis! Interesting article here. Although this device no longer seems to be sold by Digium, my shop has been looking at picking up an AA50 from a third party due to its lower cost than the AA60 and (in theory) better reliability, due to being a diskless system.

    These issues you bring up are pretty disconcerting, have you had any luck getting them resolved since posting this article? Has Digium released a firmware update for these issues? The company seems to be trying to get away from this product as fast as possible (another warning sign, it would seem). However, they might just be trying to push sales of the AA60, which appears to be a disk-based device at twice the price.

    Any advice for a small office trying to get their feet wet with a low-impact / low-maintenance VoIP PBX?

  5. Hi Reggie,

    Our solution to the problem has been to reboot the AA50 at the beginning of each day, but you have to make sure the unit is not writing voicemail or you will face file system corruption. There is a new version of firmware available on Digium’s beta site, but the release notes do not address this issue specifically; I am loathe to update the firmware and have other issues crop up.

    At the end of the day, I would recommend using a compact computer equipped with a high efficiency power supply, whatever telephony card you need for analog service and a redundant (RAID1) disk setup to protect from disk failure. I have priced some units in recent months and you could build two units (one for standby in case of failure) for the cost of one AA50 with a year of support. If you are not well-versed in telephony solutions, I also recommend engaging an experienced party to help with all of the planning and implementation.

    On a side note, you will want to check with Digium to ensure that the license is transferable to you if you buy used. Otherwise, you may end up with a unit that you cannot get support or updated software for, and at worst, can not legally use.

    Best of luck in your search and integration!

  6. I’ve been using an AA50 for a while now and must say it definitely serves its purpose as a simple device for a small office. Other than that, i wouldn’t recommend this thing. Digium support is slow to respond (4 days to an email is common), and innadequate. Apparently they are actively working on the CF card crash issue. It’s been 5 months and they are “actively” working on it? that’s gotta be a cruel joke.

  7. First of all, I would like to commend you on your article. It’s the best AA50 support I’ve gotten EVER from Digium. You’ve pretty much hit the nail on the head. We’ve been having problems since day 1 and have not had a resolution. For a diskless PBX, I can’t believe it’s more unstable than our whitebox configuration. The web GUI doesn’t work perfectly under a different browser. No one at Digium admits to these problems here. Their resolution is for me to continue to wait for a resolution while my system reboots and voicemails get full and file system gets corrupted. Last time I checked, a pbx is usually one of the most stable part of the IT infrastructure. It’s nice to see someone doing a great job of explaining their issues. I will never buy a Digium product again.

  8. […] Background: http://www.keycruncher.com/blog/2009/11/02/digium-confirms-major-issues-with-aa50-voip-appliance-spo… […]

  9. I’ve been waiting for 4 months, they still have not posted an update. Digium can go @*$% themselves for the terrible product, support, and pricing they provide. For a grand i’d expect this thing to work.

    They send all of their basic support to the evening group which only responds at night, so if you have to reply back it wont get to them for another 24 hours. And their first response doesn’t normally come for several days.

    HORRIBLE product HORRIBLE support.

    I’m with Kong I will never give digium another dollar. BTW I run an entire call center on Asterisk’s open source version and now believe even more that open source is the way to go.

  10. Has anyone come to a resolution or heard any more about the CF read-only problem? I have been struggling to get this fixed for weeks now. Thanks.

  11. Does anyone know how this product would perform in a 50+ seat call center with an additional 50 office personnel? I’m trying to decide whether or not to go with a traditional Toshiba or Avaya system or to go with Asterisk.
    Any information would be helpful!
    Thanks.

  12. Hi Francesca,

    Asterisk would work well for what little requirements you mention here, but the AA50 from Digium most certainly would not!

    There is an Asterisk users group on LinkedIn where you may want to ask questions too.

    -Dennis

  13. […] Background: http://www.keycruncher.com/blog/2009/11/02/digium-confirms-major-issues-with-aa50-voip-appliance-spo… […]

Leave a Reply

You must be logged in to post a comment.