Follow-up: Digium AA50 Reboot Issues — A Temporary Work-around

This is a follow-up my original blog post here: Digium Confirms Major Issues with AA50 VoIP Appliance: Spotaneous Reboots and Memory Card Write-Lock, A Review

Since I originally posted, I seem to have found a band-aid fix that is getting our AA50 through the days without spontaneous reboots and dropped calls in the process. I thought others might find this helpful too. Although not ideal, I am currently rebooting the AA50 shortly before it starts seeing steady use each day and thus far (knock on wood), the unit has not rebooted unexpectedly; we are going on 5 days since I put this reboot process in place.

A little bit of background before I present my work-around: The unit in question utilizes a custom “office closed” greeting that is played before all other greetings on off-hours. We deliberately set up the system to clear all of these types of messages (we call them “pre-greetings”) after a reboot. Accordingly, if the unit reboots and the office is not open, callers may not know that fact and expect a quick reply back. In order to minimize the number of cases where someone calls the office and does not hear the “office closed” pre-greeting, I reboot the phone system each Monday-Friday at 8:40 am. The office opens at 9:00 am and the receptionist routinely arrives early to listen to messages before the day starts; it is important for everything to be rebooted and ready when she arrives.

Although I do have OpenVPN access to this network, SSH with a strong password direct to the box is how this example is presented as I realize many of you might not have this luxury. I recommend that you port-forward some port other than 22 to prevent the most basic of attacks and most certainly use a strong password; if possible, restrict your origination networks as well. I based my solution on the expect package for Linux and it is working quite well.

A note on syntax: In the below examples, I use brackets [ ] to contain arguments that you will need to change to match your particular situation. For example, if your AA50’s password is 1s3curedb0x!, you would replace “[your_password]” with “1s3curedb0x!” in the below examples.

A note on requirements: My examples require the expect package and assume you are using Linux for this exercise. On a stock install of Ubuntu, you can simply “sudo apt-get install expect” to get what you need.

The reboot script: This is the main reboot script that gets called each morning. This script will remotely connect to an AA50, gracefully restart Asterisk (i.e. do not reboot in the middle of a call) and then immediately reboot the AA50.

reboot.sh script is available here

Automated log move and download scripts: If you have full logging enabled on the AA50, it does not get saved long-term so far as I can tell; it appears that the log is only stored in volatile RAM which is cleared on each reboot. Accordingly, I download my debug logs just before rebooting. I assume that if you have an AA50, you probably do not have a super high-trafficked system, but do be aware that the way I do this leaves behind events that you will not have logged right around the few seconds that it takes to login to the unit, download the log and then reboot the AA50.

Also note that this would be a much better solution if I did everything in one script. Alas, I did this piecemeal and have not yet compiled everything into one script. If you would like to do so and post it for everyone else to see, please do, but I do not have time at the moment.

move_full.sh script (moves the last full log downloaded to a file with the timestamp  of now before we download the newest log to replace it):

move_full.sh.txt is available here

scp_download.sh (download the current full log from the remote AA50 to our local machine before we reboot):

scp_download.sh.txt is available here

The cron job: This is really simple, and if you have never set up a cron job before, do a search for “crontab generator” and you will find all of the help you need.

#call the following three scripts every M-F @ 0840
40 8 * * 1-5 move_full.sh ; scp_download.sh ; reboot.sh

There you have it. It is not pretty, but it works for me. If you have a similar problem with your AA50 and are not sure how to remedy the problem yourself, please get in touch with me at http://DennisLittle.com and I will be happy to help.

Disclaimer: I am not a programmer, so use this at your own risk.

5 Responses to “Follow-up: Digium AA50 Reboot Issues — A Temporary Work-around”

  1. Installing shared SSH key will result in passwordless login, expect package is a complexity that you don’t need.

    ssh @asterisk.server “asterisk -rx ‘restart gracefully'”

    Also my aa50 doesn’t have any problems, maybe you should get digium to fix it.

    Good luck.

  2. Thanks for sharing all this information. I have experienced the same issues as mentioned in your posts. This has allowed me to “band-aid” my AA50 install as well. Also, I wanted to mention that a Digium tech did indicate in an e-mail that the base AA50 system was being upgraded, but isn’t expected to be released for months. Hopefully this base upgrade will resolve these particularly major issues. Again; thanks for the invaluable information to keep my AA50 in a closer to production state!

  3. Thanks for sharing this information, we too have been experienced these issues. We would like to implement this procedure, but our AA50 will accept the cron job but it never triggers the script.. By looking into the log (/var/log/messages) file, we see the call to the script but we always get this message “crond couldn’t fork(), user root”. We added the cron job and created the script with as root , so It shouldn’t be a permissions issue. Do you have any idea why our cron job does not execute the script?
    20 14 * * * /usr/script.sh
    nor
    20 14 * * * echo “save this” > test.log
    Thanks in advance.

  4. Hi Oscar,

    I wonder if you are using the latest firmware for the AA50, v2.0.0.6?

    We recently upgraded to this version and the problem seems to be titrated quite a lot. We have not been running this version long enough in the field to tell if the random reboot issue completely went away, but I can tell you that the system is no longer spontaneously rebooting at 1-2 day intervals.

    Regarding cron on the AA50:

    I have never been able to get cron on the AA50 to behave properly and I am unsure if this is a AA50 or BusyBox limitation; I am unsure that cron is intended to work on the AA50. Most of the filesystem on the AA50 is volatile, meaning that it is reset to a fixed state upon reboot. To be honest, I have never spent enough time working on the AA50’s cron capabilities to know if it stores crontab permanently on the flash or in the volatile RAM. If the case is the latter, your crontab would be erased after the first reboot.

    In my case, I already had another Linux computer remote to the AA50 and that is where the cron job to reboot the AA50 was running from. I needed some advanced functionality I was not sure BusyBox would support as well, so it made sense for my particular install.

    I would try out the latest firmware though (2.0.0.6 from Digium) and see if that does not resolve your problem.

    One point of note: The system I upgraded was no longer under service contract with Digium when the firmware was released. Because I had the problem from the very beginning, Digium agreed to give me the firmware without a support contract, but they could not offer any further support without starting a new contract. I knew what I was doing on the system, so it was a fair compromise for me.

    If you find you need a hand getting upgraded, please feel free to ping me.

    -Dennis

  5. Can you share your trick for using the lan on aa50.

Leave a Reply

You must be logged in to post a comment.