Caching mechanism mem overconsumption

[Dayman]

Has anyone noticed memory overconsumption since 3.0.7 (most likely introduced by this commit https://github.com/qbittorrent/qBittorr ... a6044dcab2 ) when disk write cache size is set to auto.

For example with manual 64 Meg cache i get around 160 Megs of ram consumed. With disk cache set to auto this goes to around 600+ Megs.
This would be fine if, for example, I'm downloading with high speed and some heavy disk I/O is running in the background. But this happens even if I have no downloads at all - only uploads.

P.S. That's probably libtorrent bug, but nevertheless I'd like somebody to confirm this.

[ironcross]

Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.

[Dayman]

[quote="ironcross"]
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
[/quote]
Auto was introduced in 3.0.6 (set write cache to 0).

[ironcross]

I use 3.0.6 too. No such thing. Value 0 can not be set; it's only from 1 and above. If you try 0 qB automatically reverts the last value.

[Reith]

I've noticed qBT seems to have a problem with memory consumption slowly creeping upwards the longer I leave it open, regardless of what it's doing.

It uses several times what I used to experience in uTorrent, which I was able to leave running for weeks at a time without running into this issue.  Here's an attachment of how it looks on my system:

[tekko]

[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]

https://github.com/qbittorrent/qBittorr ... nt-9209554

For us, it would be 3.0.7.

[Dayman]

[quote="tekko"]
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]

https://github.com/qbittorrent/qBittorr ... nt-9209554

For us, it would be 3.0.7.
[/quote]
Damn, too early for a report it seems. Will have to wait for 3.0.7 to be publicly available then.

[tekko]

Could setting "sessionSettings.cache_size" be the problem? It seems to handle the read cache also.

http://www.rasterbar.com/products/libto ... anual.html

"cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).
...
use_read_cache, is set to true (default), the disk cache is also used to cache pieces read from disk. Blocks for writing pieces takes presedence."

[sledgehammer_999]

From the libtorrent API documentation:

cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).

Disk buffers are allocated using a pool allocator, the number of blocks that are allocated at a time when the pool needs to grow can be specified in cache_buffer_chunk_size. This defaults to 16 blocks. Lower numbers saves memory at the expense of more heap allocations. It must be at least 1.

Emphasis mine. When the user selects "0(auto)" the cache_size variable is set to -1. I think this commit should be partially reverted.

[Dayman]

I've reported this on qBt bugtracker: https://github.com/qbittorrent/qBittorrent/issues/258

So this is how I see this. LT uses the cache size value for write, read cache and network buffers. And what happens looks to me like this (using libastral w/o reading LT code):

1. qBt starts; usual warm-up period
2. after some time we hit cache limit
3. old cache items are evicted only when we hit cache limit
4. we stay with cache size == cache limit size forever

Well, looks like LT has no periodical GC of any sorts.

[Tomaso]

[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]

[Dayman]

[quote="Tomaso"]
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
[/quote]
I'll try aio HEAD then.

---

Doesn't build w/ bufs_size redefinition. I'll leave it at that and wait for 0.16.6 release :S

[Nemo]

The devs at libtorrent are trying to improve on this problem I see, thats good of course .