ClamWin Free Antivirus Forum Index
ClamWin Free Antivirus
Support and Discussion Forums
Reply to topic
main cld/cvd file size
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
Hi, in the past I found strange things with the file size of main.cld/cvd (another question should be why 2 extensions Confused depending of if it is updated or not, or whatever) and I thought it was solved but after months without checking, looks the problem persists.

The "problem" is that the size of the file keeps growing, growing, and growing.

I understand that it is a normal behavior of every antivirus, to increase the database file size with each update, but it is annoying that, if you delete or rename the file and, then, so, you "trick" ClamWin so it asks for the need to download the database, the new main.clv/cvd file that it downloads, has a size half of the one you just deleted or renamed.

This is just disturbing, but most disturbing is when you see in the update log window (or file) that, even the difference of size, it is reported the same signatures in the database file.

From a normal update the main file was 73.8MB (for normal I mean updated every time without renaming or deleting since clamwin install)

From a forced update the main file was 29.3MB

How is this possible? Why is this? Is it a bug? What file size should we trust? What should we trust at all? Are we equally safe with a database file of 73.8MB and 29.3MB?

Here is posted the log from the normal update and the forced update for download a new main file:
Quote:
ClamAV update process started at Sun Oct 16 00:35:04 2011
main.cld is up to date (version: 54, sigs: 1044387, f-level: 60, builder: sven)
Downloading daily-13793.cdiff [100%]
Downloading daily-13794.cdiff [100%]
Downloading daily-13795.cdiff [100%]
Downloading daily-13796.cdiff [100%]
Downloading daily-13797.cdiff [100%]
Downloading daily-13798.cdiff [100%]
Downloading daily-13799.cdiff [100%]
Downloading daily-13800.cdiff [100%]
Downloading daily-13801.cdiff [100%]
Downloading daily-13802.cdiff [100%]
Downloading daily-13803.cdiff [100%]
Downloading daily-13804.cdiff [100%]
Downloading daily-13805.cdiff [100%]
daily.cld updated (version: 13805, sigs: 11065, f-level: 60, builder: guitar)
bytecode.cld is up to date (version: 147, sigs: 38, f-level: 60, builder: edwin)
Database updated (1055490 signatures) from clamav.n2000.net (IP: 80.80.88.40)
--------------------------------------
ClamAV update process started at Sun Oct 16 00:36:05 2011
Downloading main.cvd [100%]
main.cvd updated (version: 54, sigs: 1044387, f-level: 60, builder: sven)
daily.cld is up to date (version: 13805, sigs: 11065, f-level: 60, builder: guitar)
bytecode.cld is up to date (version: 147, sigs: 38, f-level: 60, builder: edwin)
Database updated (1055490 signatures) from clamav.n2000.net (IP: 80.80.88.40)



Thanks for clear my doubts Smile
Regards.
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
I never noticed this before, but you are right. All database file sizes are changed quite a bit if you delete the files and then update the database again. In my case, the main file went from 75,639 KB to 30,030 KB, the daily file went from 801 KB to 286 KB, and the bytecode filesize went from 252 KB to 50 KB. Clam is supposed to have incremental updates, so I can only surmise that there is some "houskeeping" data from the updates that is included in the files until you compact it by manually updating.

Perhaps someone else can chime in here with the answer. I wouldn't worry about it as long as the number of signatures don't change during this process, and they did not seem to in my experiment. The updates are supposed to be digitally-signed, so the updating process is pretty safe. I wouldn't worry too much about it, but it would be nice to know the reason, eh?

Regards,
View user's profileSend private message
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
GuitarBob wrote:
but it would be nice to know the reason, eh?


Yep, that is where I'm pointing to Smile

And thanks for confirm it.

Let's wait for the devs/admins explanation Smile
View user's profileSend private message
johne155


Joined: 28 Sep 2009
Posts: 4
Location: Indiana, USA
Reply with quote
The current main.cvd is bigger than the previous main.cvd as it contains all the definitions that had been in the daily.cvds since the last main was released.

BUT when the old main.CVD is "updated" and becomes main.CLD it is not done in a very efficient manner, it is "patched", so to speak, with the difference between the old and the new, but not very well, so it it much, much larger.

If you download the new main.CVD directly, instead of patching your old main.CVD into the new main.CLD, it as tight as they can make it.

The same thing happens each day with the dailies.
If you directly download a new copy of daily.CVD is is very small, but as soon as it is updated it is unpacked updated and repacked as a much larger daily.CLD

Simply put, the full/fresh downloaded .CVDs are always smaller then the updated versions, which become .CLDs

John
View user's profileSend private message
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
And that is very understandable, of course. But for me is like a bug.

Instead just append the new updates, why not decompress, merge and re-compress? (I think the databse is compressed, but I haven't found which compressor uses)

I know, you might be thinking now that I'm a disk space paranoid (that I am Razz) and the space needs growth is minimal, but, why waste space?

In previous versions, one of the updates said that improved a faster loading of the database, so the updates could be merged instead appended. For me has more sense. While you keep a minimal database, the functionality is not degraded.



Before post the above I searched I little here and looks like http://forums.clamwin.com/viewtopic.php?p=14238#14238 freshclam.exe accept parameters through freshclam.conf...

I think we can enable compression by creating a freshclam.conf with a line like "CompressLocalDatabase yes" file and pass parameters to freshclam.exe by clamwin.conf setting "freshclamparams = --config-file=freshclam.conf".

I think I'll need to test this...
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
Please let us know how it goes. I always thought Clam AV had differential updates, but it looks like this isn't entirely true.

Regards,
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
I have run this by the Clam AV team and will let you know what they say.

Regards,
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
You are correct. Here is what one of the Clam programmers says:

On 10/18/11 18:41, Robert Scroggins wrote:
> Hello All:
>
> Below is a thread from the ClamWin forum about this topic.

All the guy needs to do is setting "CompressLocalDatabase" to yes in
freshclam.conf.

Regards,
View user's profileSend private message
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
I have done a backup of the db folder (to test more than once) and I'll try it. Now there aren't database updates and I suppose that the compression parameter is only executed when there is an update to the files.

Right now, from what I read in another thread this afternoon, I only left clamwin.conf "freshclamparams" as "freshclamparams = CompressLocalDatabase yes". If it doesn't work, I'll try as what I posted above, calling for freshclam.conf with "CompressLocalDatabase yes" inside.
View user's profileSend private message
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
Ok, now there was an update for daily file.

I tried the following methods:

1.- Adding the line "freshclamparams = CompressLocalDatabase yes" to clamwin.conf: result in an append of ~1KB to daily.cld

2.- forgetting the freshclamparams in clamwin.conf and creating a freshclam.conf with the line "CompressLocalDatabase yes": result in an append of ~1KB to daily.cld

3.- calling freshclam.conf with the line "CompressLocalDatabase yes" through clamwin.conf with the line "freshclamparams = --config-file=freshclam.conf": result in an append of ~1KB to daily.cld

4.- calling freshclam.conf with the line "CompressLocalDatabase yes" through clamwin.conf with the line "freshclamparams = config-file=freshclam.conf" (I removed the two dashes): result in an append of ~1KB to daily.cld

A full update (deleting database files), and without any modification to conf files, gave a daily.cld file of ~600KB less (by the way, the same size as if I compress it with Winrar).

At my eyes, looks that doesn't happen anything. Confused If it is supposed to be compressed the file to equal the new full download... that doesn't happen. Indeed, the downloaded update is appended :/

Probably I'm missing something... Laughing but is frustrating also Rolling Eyes
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
Perhaps you need to re-start ClamWin after making the configuration change. It may still be working off of the configuration it had when the computer was started before the change.

Regards,
View user's profileSend private message
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
GuitarBob wrote:
Perhaps you need to re-start ClamWin after making the configuration change. It may still be working off of the configuration it had when the computer was started before the change.

Regards,


I do restart ClamWin between each attempt Wink I also don't use ClamTray, so easier to restart Smile

The only thing I left to do is to run freshclam.exe stand alone, without invoking it from the ClamWin GUI, but that involves a full fill of freshclam.conf with the options of the thread I posted. I only gave a short try this afternoon, this way, and you have to manually set the path to the db, the mirror, and other options. But I left this method because we are getting out of the concept of just update. I mean that you'll need to use freshclam to update the database separately of ClamWin itself, that is not bad, but complicating the update process just to add compression to the database :/

But if it is the only way... maybe should be something to consider, but then, maybe, would be easier to delete the db files and download it from scratch.
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
It's been a couple of days now since I added the CompressLocalDatabase = yes to my ClamWin.conf file (at the end of the updates section), and the problem seems to have been fixed. It was not fixed immediately, but I noticed there is now no change in the main db size after I deleted the files and downloaded them again. I only noticed the main db size--I did not note the other db files. I suppose it is automatically compressed on some sort of schedule--daily perhaps.

Try it and see if you agree after a day or so.

Regards,
View user's profileSend private message
blgd


Joined: 08 May 2007
Posts: 26
Reply with quote
Ok, let's see Smile

P.S.: you meant the parameter standalone or inside "freshclamparams"? Post the line to see how you wrote it Wink
View user's profileSend private message
GuitarBob


Joined: 09 Jul 2006
Posts: 4376
Location: USA
Reply with quote
In the ClamWin.conf file at the end of the Updates section:

[Updates]
checkversion = 1
checkversionurl = http://clamwin.sourceforge.net/clamwinver.php
warnoutofdate = 1
enable = 1
dbupdatelogfile = C:\ProgramData\.clamwin\log\ClamUpdateLog.txt
dbmirror = database.clamav.net
updateonlogon = 0
frequency = Daily
weekday = 2
time = 11:56:00
CompressLocalDatabase = yes

Regards,
View user's profileSend private message
main cld/cvd file size
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
All times are GMT  
Page 1 of 2  

  
  
 Reply to topic