Page 1 of 1

Caching mechanism mem overconsumption

Posted: Fri Oct 19, 2012 3:38 pm
by Dayman
Has anyone noticed memory overconsumption since 3.0.7 (most likely introduced by this commit https://github.com/qbittorrent/qBittorr ... a6044dcab2 ) when disk write cache size is set to auto.

For example with manual 64 Meg cache i get around 160 Megs of ram consumed. With disk cache set to auto this goes to around 600+ Megs.
This would be fine if, for example, I'm downloading with high speed and some heavy disk I/O is running in the background. But this happens even if I have no downloads at all - only uploads.

P.S. That's probably libtorrent bug, but nevertheless I'd like somebody to confirm this.

Re: Caching mechanism mem overconsumption

Posted: Fri Oct 19, 2012 4:10 pm
by ironcross
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.

Re: Caching mechanism mem overconsumption

Posted: Fri Oct 19, 2012 6:15 pm
by Dayman
[quote="ironcross"]
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
[/quote]
Auto was introduced in 3.0.6 (set write cache to 0).

Re: Caching mechanism mem overconsumption

Posted: Fri Oct 19, 2012 10:48 pm
by ironcross
I use 3.0.6 too. No such thing. Value 0 can not be set; it's only from 1 and above. If you try 0 qB automatically reverts the last value.

Re: Caching mechanism mem overconsumption

Posted: Sat Oct 20, 2012 12:25 am
by Reith
I've noticed qBT seems to have a problem with memory consumption slowly creeping upwards the longer I leave it open, regardless of what it's doing.

It uses several times what I used to experience in uTorrent, which I was able to leave running for weeks at a time without running into this issue.  Here's an attachment of how it looks on my system:

Re: Caching mechanism mem overconsumption

Posted: Sat Oct 20, 2012 6:16 am
by tekko
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]

https://github.com/qbittorrent/qBittorr ... nt-9209554

For us, it would be 3.0.7.

Re: Caching mechanism mem overconsumption

Posted: Sat Oct 20, 2012 9:23 am
by Dayman
[quote="tekko"]
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]

https://github.com/qbittorrent/qBittorr ... nt-9209554

For us, it would be 3.0.7.
[/quote]
Damn, too early for a report it seems. Will have to wait for 3.0.7 to be publicly available then.

Re: Caching mechanism mem overconsumption

Posted: Thu Nov 29, 2012 1:47 am
by tekko
Could setting "sessionSettings.cache_size" be the problem? It seems to handle the read cache also.

http://www.rasterbar.com/products/libto ... anual.html

"cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).
...
use_read_cache, is set to true (default), the disk cache is also used to cache pieces read from disk. Blocks for writing pieces takes presedence."

Re: Caching mechanism mem overconsumption

Posted: Thu Nov 29, 2012 11:58 pm
by sledgehammer_999
From the libtorrent API documentation:
cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).

Disk buffers are allocated using a pool allocator, the number of blocks that are allocated at a time when the pool needs to grow can be specified in cache_buffer_chunk_size. This defaults to 16 blocks. Lower numbers saves memory at the expense of more heap allocations. It must be at least 1.
Emphasis mine. When the user selects "0(auto)" the cache_size variable is set to -1. I think this commit should be partially reverted.

Re: Caching mechanism mem overconsumption

Posted: Fri Nov 30, 2012 4:41 pm
by Dayman
I've reported this on qBt bugtracker: https://github.com/qbittorrent/qBittorrent/issues/258

So this is how I see this. LT uses the cache size value for write, read cache and network buffers. And what happens looks to me like this (using libastral w/o reading LT code):
  1. qBt starts; usual warm-up period
  2. after some time we hit cache limit
  3. old cache items are evicted only when we hit cache limit
  4. we stay with cache size == cache limit size forever
Well, looks like LT has no periodical GC of any sorts.

Re: Caching mechanism mem overconsumption

Posted: Sat Dec 01, 2012 10:51 am
by Tomaso
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]

Re: Caching mechanism mem overconsumption

Posted: Sat Dec 01, 2012 12:01 pm
by Dayman
[quote="Tomaso"]
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
[/quote]
I'll try aio HEAD then.
Doesn't build w/ bufs_size redefinition. I'll leave it at that and wait for 0.16.6 release :S

Re: Caching mechanism mem overconsumption

Posted: Sat Dec 01, 2012 1:53 pm
by Nemo
The devs at libtorrent are trying to improve on this problem I see, thats good of course :).