Hi, in the past I found strange things with the file size of main.cld/cvd (another question should be why 2 extensions depending of if it is updated or not, or whatever) and I thought it was solved but after months without checking, looks the problem persists.

The "problem" is that the size of the file keeps growing, growing, and growing.

I understand that it is a normal behavior of every antivirus, to increase the database file size with each update, but it is annoying that, if you delete or rename the file and, then, so, you "trick" ClamWin so it asks for the need to download the database, the new main.clv/cvd file that it downloads, has a size half of the one you just deleted or renamed.

This is just disturbing, but most disturbing is when you see in the update log window (or file) that, even the difference of size, it is reported the same signatures in the database file.

From a normal update the main file was 73.8MB (for normal I mean updated every time without renaming or deleting since clamwin install)

From a forced update the main file was 29.3MB

How is this possible? Why is this? Is it a bug? What file size should we trust? What should we trust at all? Are we equally safe with a database file of 73.8MB and 29.3MB?

Here is posted the log from the normal update and the forced update for download a new main file:


Thanks for clear my doubts
Regards.

I never noticed this before, but you are right. All database file sizes are changed quite a bit if you delete the files and then update the database again. In my case, the main file went from 75,639 KB to 30,030 KB, the daily file went from 801 KB to 286 KB, and the bytecode filesize went from 252 KB to 50 KB. Clam is supposed to have incremental updates, so I can only surmise that there is some "houskeeping" data from the updates that is included in the files until you compact it by manually updating.

Perhaps someone else can chime in here with the answer. I wouldn't worry about it as long as the number of signatures don't change during this process, and they did not seem to in my experiment. The updates are supposed to be digitally-signed, so the updating process is pretty safe. I wouldn't worry too much about it, but it would be nice to know the reason, eh?

Regards,

Yep, that is where I'm pointing to

And thanks for confirm it.

Let's wait for the devs/admins explanation

The current main.cvd is bigger than the previous main.cvd as it contains all the definitions that had been in the daily.cvds since the last main was released.

BUT when the old main.CVD is "updated" and becomes main.CLD it is not done in a very efficient manner, it is "patched", so to speak, with the difference between the old and the new, but not very well, so it it much, much larger.

If you download the new main.CVD directly, instead of patching your old main.CVD into the new main.CLD, it as tight as they can make it.

The same thing happens each day with the dailies.
If you directly download a new copy of daily.CVD is is very small, but as soon as it is updated it is unpacked updated and repacked as a much larger daily.CLD

Simply put, the full/fresh downloaded .CVDs are always smaller then the updated versions, which become .CLDs

John

And that is very understandable, of course. But for me is like a bug.

Instead just append the new updates, why not decompress, merge and re-compress? (I think the databse is compressed, but I haven't found which compressor uses)

I know, you might be thinking now that I'm a disk space paranoid (that I am ) and the space needs growth is minimal, but, why waste space?

In previous versions, one of the updates said that improved a faster loading of the database, so the updates could be merged instead appended. For me has more sense. While you keep a minimal database, the functionality is not degraded.



Before post the above I searched I little here and looks like https://forums.clamwin.com/viewtopic.php?p=14238#14238 freshclam.exe accept parameters through freshclam.conf...

I think we can enable compression by creating a freshclam.conf with a line like "CompressLocalDatabase yes" file and pass parameters to freshclam.exe by clamwin.conf setting "freshclamparams = --config-file=freshclam.conf".

I think I'll need to test this...

Please let us know how it goes. I always thought Clam AV had differential updates, but it looks like this isn't entirely true.

Regards,

I have run this by the Clam AV team and will let you know what they say.

Regards,

You are correct. Here is what one of the Clam programmers says:

On 10/18/11 18:41, Robert Scroggins wrote:
> Hello All:
>
> Below is a thread from the ClamWin forum about this topic.

All the guy needs to do is setting "CompressLocalDatabase" to yes in
freshclam.conf.

Regards,

I have done a backup of the db folder (to test more than once) and I'll try it. Now there aren't database updates and I suppose that the compression parameter is only executed when there is an update to the files.

Right now, from what I read in another thread this afternoon, I only left clamwin.conf "freshclamparams" as "freshclamparams = CompressLocalDatabase yes". If it doesn't work, I'll try as what I posted above, calling for freshclam.conf with "CompressLocalDatabase yes" inside.

Ok, now there was an update for daily file.

I tried the following methods:

1.- Adding the line "freshclamparams = CompressLocalDatabase yes" to clamwin.conf: result in an append of ~1KB to daily.cld

2.- forgetting the freshclamparams in clamwin.conf and creating a freshclam.conf with the line "CompressLocalDatabase yes": result in an append of ~1KB to daily.cld

3.- calling freshclam.conf with the line "CompressLocalDatabase yes" through clamwin.conf with the line "freshclamparams = --config-file=freshclam.conf": result in an append of ~1KB to daily.cld

4.- calling freshclam.conf with the line "CompressLocalDatabase yes" through clamwin.conf with the line "freshclamparams = config-file=freshclam.conf" (I removed the two dashes): result in an append of ~1KB to daily.cld

A full update (deleting database files), and without any modification to conf files, gave a daily.cld file of ~600KB less (by the way, the same size as if I compress it with Winrar).

At my eyes, looks that doesn't happen anything. If it is supposed to be compressed the file to equal the new full download... that doesn't happen. Indeed, the downloaded update is appended :/

Probably I'm missing something... but is frustrating also

Perhaps you need to re-start ClamWin after making the configuration change. It may still be working off of the configuration it had when the computer was started before the change.

Regards,

I do restart ClamWin between each attempt I also don't use ClamTray, so easier to restart

The only thing I left to do is to run freshclam.exe stand alone, without invoking it from the ClamWin GUI, but that involves a full fill of freshclam.conf with the options of the thread I posted. I only gave a short try this afternoon, this way, and you have to manually set the path to the db, the mirror, and other options. But I left this method because we are getting out of the concept of just update. I mean that you'll need to use freshclam to update the database separately of ClamWin itself, that is not bad, but complicating the update process just to add compression to the database :/

But if it is the only way... maybe should be something to consider, but then, maybe, would be easier to delete the db files and download it from scratch.

It's been a couple of days now since I added the CompressLocalDatabase = yes to my ClamWin.conf file (at the end of the updates section), and the problem seems to have been fixed. It was not fixed immediately, but I noticed there is now no change in the main db size after I deleted the files and downloaded them again. I only noticed the main db size--I did not note the other db files. I suppose it is automatically compressed on some sort of schedule--daily perhaps.

Try it and see if you agree after a day or so.

Regards,

Ok, let's see

P.S.: you meant the parameter standalone or inside "freshclamparams"? Post the line to see how you wrote it

In the ClamWin.conf file at the end of the Updates section:

[Updates]
checkversion = 1
checkversionurl = https://clamwin.sourceforge.net/clamwinver.php
warnoutofdate = 1
enable = 1
dbupdatelogfile = C:\ProgramData\.clamwin\log\ClamUpdateLog.txt
dbmirror = database.clamav.net
updateonlogon = 0
frequency = Daily
weekday = 2
time = 11:56:00
CompressLocalDatabase = yes

Regards,