Caching mechanism mem overconsumption
Caching mechanism mem overconsumption
Has anyone noticed memory overconsumption since 3.0.7 (most likely introduced by this commit https://github.com/qbittorrent/qBittorr ... a6044dcab2 ) when disk write cache size is set to auto.
For example with manual 64 Meg cache i get around 160 Megs of ram consumed. With disk cache set to auto this goes to around 600+ Megs.
This would be fine if, for example, I'm downloading with high speed and some heavy disk I/O is running in the background. But this happens even if I have no downloads at all - only uploads.
P.S. That's probably libtorrent bug, but nevertheless I'd like somebody to confirm this.
For example with manual 64 Meg cache i get around 160 Megs of ram consumed. With disk cache set to auto this goes to around 600+ Megs.
This would be fine if, for example, I'm downloading with high speed and some heavy disk I/O is running in the background. But this happens even if I have no downloads at all - only uploads.
P.S. That's probably libtorrent bug, but nevertheless I'd like somebody to confirm this.
Last edited by Dayman on Sat Oct 20, 2012 9:21 am, edited 1 time in total.
Re: Caching mechanism mem overconsumption
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
Last edited by ironcross on Fri Oct 19, 2012 4:14 pm, edited 1 time in total.
Re: Caching mechanism mem overconsumption
[quote="ironcross"]
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
[/quote]
Auto was introduced in 3.0.6 (set write cache to 0).
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
[/quote]
Auto was introduced in 3.0.6 (set write cache to 0).
Re: Caching mechanism mem overconsumption
I use 3.0.6 too. No such thing. Value 0 can not be set; it's only from 1 and above. If you try 0 qB automatically reverts the last value.
Re: Caching mechanism mem overconsumption
I've noticed qBT seems to have a problem with memory consumption slowly creeping upwards the longer I leave it open, regardless of what it's doing.
It uses several times what I used to experience in uTorrent, which I was able to leave running for weeks at a time without running into this issue. Here's an attachment of how it looks on my system:
It uses several times what I used to experience in uTorrent, which I was able to leave running for weeks at a time without running into this issue. Here's an attachment of how it looks on my system:
Re: Caching mechanism mem overconsumption
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]
https://github.com/qbittorrent/qBittorr ... nt-9209554
For us, it would be 3.0.7.
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]
https://github.com/qbittorrent/qBittorr ... nt-9209554
For us, it would be 3.0.7.
Re: Caching mechanism mem overconsumption
[quote="tekko"]
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]
https://github.com/qbittorrent/qBittorr ... nt-9209554
For us, it would be 3.0.7.
[/quote]
Damn, too early for a report it seems. Will have to wait for 3.0.7 to be publicly available then.
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]
https://github.com/qbittorrent/qBittorr ... nt-9209554
For us, it would be 3.0.7.
[/quote]
Damn, too early for a report it seems. Will have to wait for 3.0.7 to be publicly available then.
Re: Caching mechanism mem overconsumption
Could setting "sessionSettings.cache_size" be the problem? It seems to handle the read cache also.
http://www.rasterbar.com/products/libto ... anual.html
"cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).
...
use_read_cache, is set to true (default), the disk cache is also used to cache pieces read from disk. Blocks for writing pieces takes presedence."
http://www.rasterbar.com/products/libto ... anual.html
"cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).
...
use_read_cache, is set to true (default), the disk cache is also used to cache pieces read from disk. Blocks for writing pieces takes presedence."
-
- Administrator
- Posts: 2443
- Joined: Sun Jan 23, 2011 1:17 pm
Re: Caching mechanism mem overconsumption
From the libtorrent API documentation:
Emphasis mine. When the user selects "0(auto)" the cache_size variable is set to -1. I think this commit should be partially reverted.cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).
Disk buffers are allocated using a pool allocator, the number of blocks that are allocated at a time when the pool needs to grow can be specified in cache_buffer_chunk_size. This defaults to 16 blocks. Lower numbers saves memory at the expense of more heap allocations. It must be at least 1.
Re: Caching mechanism mem overconsumption
I've reported this on qBt bugtracker: https://github.com/qbittorrent/qBittorrent/issues/258
So this is how I see this. LT uses the cache size value for write, read cache and network buffers. And what happens looks to me like this (using libastral w/o reading LT code):
So this is how I see this. LT uses the cache size value for write, read cache and network buffers. And what happens looks to me like this (using libastral w/o reading LT code):
- qBt starts; usual warm-up period
- after some time we hit cache limit
- old cache items are evicted only when we hit cache limit
- we stay with cache size == cache limit size forever
Re: Caching mechanism mem overconsumption
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
Re: Caching mechanism mem overconsumption
[quote="Tomaso"]
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
[/quote]
I'll try aio HEAD then.
Doesn't build w/ bufs_size redefinition. I'll leave it at that and wait for 0.16.6 release :S
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
[/quote]
I'll try aio HEAD then.
Doesn't build w/ bufs_size redefinition. I'll leave it at that and wait for 0.16.6 release :S
Last edited by Dayman on Sat Dec 01, 2012 12:24 pm, edited 1 time in total.
Re: Caching mechanism mem overconsumption
The devs at libtorrent are trying to improve on this problem I see, thats good of course .