Postfix + ClamAV + SpamAssassin + Amavisd-New

As we all know, ClamAV versions older than 0.95 stopped working yesterday. This was announced some time ago by ClamAV, but caught most people by surprise when mail flow stopped cold on our email servers. ClamAV pushed down an update that broke older versions, forcing sysadmins to either disable Antivirus, or update ClamAV to a more recent version.

We were running a very old version of ClamAV, 0.91.2-1~volatile1, from 2007. An upgrade was long overdue anyway. Here are 2 ways to fix it.

Symptoms:
Email delivery is stalled. Both incoming and outgoing emails get “Stuck” on the email server.

The mail log and clamAV logs display errors similar to those below:

vim /var/log/mail.err

Apr 15 22:32:16 smtp amavis[4491]: (04491-02) (!!)WARN: all primary virus scanners failed, considering backups

Apr 15 22:32:23 smtp amavis[4491]: (04491-02) (!!)run_av (ClamAV-clamscan) FAILED – unexpected exit 50, output=”…LibClamAV Warning: *** This version of the ClamAV engine is outdated. …***\nLibClamAV Warning: *** DON’T PANIC! Read http://www.clamav.net/support/faq ***\nLibClamAV Warning: ***********************************************************\nLibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later. For more information see www.clamav.net/eol-clamav-094 and www.clamav.net/download (length: 169)\nLibClamAV Error: Problem parsing signature at line 742\nLibClamAV Error: Problem parsing database at line 742\nLibClamAV Error: Can’t load /var/lib/amavis/tmp/clamav-8e80726f4987852ee1d2d62697bc2b27/daily.ndb: Malformed database\nLibClamAV Error: Can’t load /var/lib/clamav//daily.cvd: Malformed database\nERROR: Malformed database”

Apr 15 22:32:23 smtp amavis[4491]: (04491-02) (!!)ClamAV-clamscan av-scanner FAILED: /usr/bin/clamscan unexpected exit 50, output=”…LibClamAV Warning: *** This version of the ClamAV engine is outdated. …***\nLibClamAV Warning: *** DON’T PANIC! Read http://www.clamav.net/support/faq ***\nLibClamAV Warning: ***********************************************************\nLibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later. For more information see www.clamav.net/eol-clamav-094 and www.clamav.net/download (length: 169)\nLibClamAV Error: Problem parsing signature at line 742\nLibClamAV Error: Problem parsing database at line 742\nLibClamAV Error: Can’t load /var/lib/amavis/tmp/clamav-8e80726f4987852ee1d2d62697bc2b27/daily.ndb: Malformed database\nLibClamAV Error: Can’t load /var/lib/clamav//daily.cvd: Malformed database\nERROR: Malformed database” at (eval 44) line 511.

Apr 15 22:32:23 smtp amavis[4491]: (04491-02) (!!)TROUBLE in check_mail: virus_scan FAILED: virus_scan: ALL VIRUS SCANNERS FAILED: ClamAV-clamd av-scanner FAILED: CODE(0x814f4f8) Too many retries to talk to /var/run/clamav/clamd.ctl (Can’t connect to UNIX socket /var/run/clamav/clamd.ctl: No such file or directory) at (eval 44) line 310. at (eval 44) line 511.; ClamAV-clamscan av-scanner FAILED: /usr/bin/clamscan unexpected exit 50, output=”…LibClamAV Warning: *** This version of the ClamAV engine is outdated. …***\nLibClamAV Warning: *** DON’T PANIC! Read http://www.clamav.net/support/faq ***\nLibClamAV Warning: ***********************************************************\nLibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later. For more information see www.clamav.net/eol-clamav-094 and www.clamav.net/download (length: 169)\nLibClamAV Error: Problem parsing signature at line 742\nLibClamAV Error: Problem parsin…

Apr 15 22:32:23 smtp amavis[4491]: (04491-02) (!!)…g database at line 742\nLibClamAV Error: Can’t load /var/lib/amavis/tmp/clamav-8e80726f4987852ee1d2d62697bc2b27/daily.ndb: Malformed database\nLibClamAV Error: Can’t load /var/lib/clamav//daily.cvd: Malformed database\nERROR: Malformed database” at (eval 44) line 511.

Apr 15 22:33:35 smtp amavis[4486]: (04486-02) (!!)ClamAV-clamd av-scanner FAILED: CODE(0x814f4f8) Too many retries to talk to /var/run/clamav/clamd.ctl (Can’t connect to UNIX socket /var/run/clamav/clamd.ctl: No such file or directory) at (eval 44) line 310. at (eval 44) line 511.

Solution:
A) Update ClamAV to a supported Version (Preferred Method)
I am running Debian Etch. These steps should work if you are running Etch. If you are running another distro you will need to adjust this (I’m sorry I can’t help you there).

Use Apt to update clamAV:

apt-get -t etch install clamav clamav-daemon clamav-freshclam

I chose to keep my existing clamav.conf file (located at /etc/clamav/clamav.conf). However, it is probably safe to let the upgrade replace it. I had to comment out several lines out of my conf file before ClamAV would start up. Running

/etc/init.d/clamav-daemon restart

and watching for the errors will show you what line(s) need to be removed/commented out. I had 4 lines to remove. Run clamav-daemon restart after commenting each line to see what the next line that needs to be removed is.

Check your clamAV version to make sure it updated:

dpkg -s clamav-daemon 

(mine reported: 0.95.3+dfsg-1~volatile1~etch2)

Check for errors in the mail log (tail -f /var/log/mail.log) and check for errors in the ClamAV log (tail -f /var/log/clamav/clamav.log). If you see “Reading databases from /var/lib/clamav” and “Database correctly reloaded” you should be all set. You can check this again after a few hours to see if ClamAV downloaded and applied new updates successfully (same entry in the log).

Check your queued mail by running:

postqueue -p

Postfix should begin to deliver these, if it does not start sending email, force delivery by running:

postqueue -f

B) Disable AntiVirus checks in Amavisd.conf (Not recommended)
This method is not preferred since it leaves you with no antivirus scanner, but it may be appropriate for some email servers. Edit /etc/amavisd.conf

vim /etc/amavisd.conf

and un-comment (remove the hash/# mark at the start) the line

#@bypass_virus_checks_maps = (1); #controls running of anti-virus code

Save and reload Amavisd-new:

amavisd-new reload

Check mail delivery by sending a test email from inside and outside of your network.

Below is a log of the unsuccessful troubleshooting we did before solving the issue by upgrading to the correct version. As you can see below, ClamAV did not actually update when we ran an apt-get upgrade, as it happened to be pinned back (specific requirement when we built these email servers several years ago).

The text below is provided for reference ONLY, and should not be used to resolve the errors above.

After some digging online, I found a few sites recommending that I delete the clam database and re-download a new fresh database. I’m not a big fan of deleting files when I don’t know exactly what they do so I’ll move them into a backup location.

I think you may or may not have this first part depending on how old your install is.

mkdir /var/lib/clamav/daily.inc-old
mv /var/lib/clamav/daily.inc/* /var/lib/clamav/daily.inc-old

mkdir /var/lib/clamav/main.inc-old
mv /var/lib/clamav/main.inc/* /var/lib/clamav/main.inc-old

mv /var/lib/clamav/daily.cvd /var/lib/clamav/daily.cvd-old
mv /var/lib/clamav/main.cvd /var/lib/clamav/main.cvd-old

and update with the new definitions:

freshclam

That seemed to get rid of most errors, however, mail still wasn’t getting through, and I kept receiving the error:

WARNING: Clamd was NOT notified: Can’t connect to clamd through /var/run/clamav/clamd.ctl

I remember seeing in the last set of errors that it was saying “This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later.” I hate upgrading software on production machines without testing first, but I saw no other options. So I upgraded the antivirus.

apt-get upgrade clamav-daemon clamav-freshclam

After upgrading, I found that I needed to make a few changes to my clamd.conf file.

vim /etc/clamav/clamd.conf

If you have any of these lines, comment them out with a # in front.

#ScanPartialMessages false
#HeuristicScanPrecedence false
#StructuredDataDetection false
#CommandReadTimeout 5
#SendBufTimeout 200
#MaxQueue 100

Unsure as to dependancies, I restarted the services that might use it.

/etc/init.d/clamav-daemon restart
/etc/init.d/clamav-freshclam restart
/etc/init.d/amavis restart
/etc/init.d/postfix restart

I sent a few test emails back and forth from my work email to an external email address, and it appears to work. Now, I’m sure there are a few emails stuck in the mail queue. So I’ll check that out.

This will tell me how many emails are stuck in the queue:

postqueue -p

— 15172 Kbytes in 399 Requests.

It looks like the smtp server has been down for a little while…
This will flush the email queue, and send any emails stuck in there:

postqueue -f

All the stuck mail has been sent, and email to and from external sites now works.
This method failed when new signatures/definitions were auto-downloaded. See above for permanent solution.

Others have this issue as well:
ClamAV Bugzilla
SlashDot sees it too.