Caching mechanism mem overconsumption

Other platforms, generic questions.
Post Reply
User avatar
Dayman
Administrator
Administrator
Posts: 343
Joined: Mon Jun 18, 2012 8:12 am

Caching mechanism mem overconsumption

Post by Dayman » Fri Oct 19, 2012 3:38 pm

Has anyone noticed memory overconsumption since 3.0.7 (most likely introduced by this commit https://github.com/qbittorrent/qBittorr ... a6044dcab2 ) when disk write cache size is set to auto.

For example with manual 64 Meg cache i get around 160 Megs of ram consumed. With disk cache set to auto this goes to around 600+ Megs.
This would be fine if, for example, I'm downloading with high speed and some heavy disk I/O is running in the background. But this happens even if I have no downloads at all - only uploads.

P.S. That's probably libtorrent bug, but nevertheless I'd like somebody to confirm this.
Attachments
qbt_mem.png
Last edited by Dayman on Sat Oct 20, 2012 9:21 am, edited 1 time in total.
Building x86_64 stuff on Windows due to Gentoo syndrome.

Jabber: [email protected]
Skype: x.dayman
Irc #[email protected]: Dayman

User avatar
ironcross
Veteran
Veteran
Posts: 433
Joined: Sun Aug 29, 2010 5:34 pm

Re: Caching mechanism mem overconsumption

Post by ironcross » Fri Oct 19, 2012 4:10 pm

Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
Last edited by ironcross on Fri Oct 19, 2012 4:14 pm, edited 1 time in total.

User avatar
Dayman
Administrator
Administrator
Posts: 343
Joined: Mon Jun 18, 2012 8:12 am

Re: Caching mechanism mem overconsumption

Post by Dayman » Fri Oct 19, 2012 6:15 pm

[quote="ironcross"]
Seeding 25 torrents, 4 of them active; 16MB Disk write cache size (default value, i don't change it and i don't see Auto option), around 24MB RAM used.
[/quote]
Auto was introduced in 3.0.6 (set write cache to 0).
Building x86_64 stuff on Windows due to Gentoo syndrome.

Jabber: [email protected]
Skype: x.dayman
Irc #[email protected]: Dayman

User avatar
ironcross
Veteran
Veteran
Posts: 433
Joined: Sun Aug 29, 2010 5:34 pm

Re: Caching mechanism mem overconsumption

Post by ironcross » Fri Oct 19, 2012 10:48 pm

I use 3.0.6 too. No such thing. Value 0 can not be set; it's only from 1 and above. If you try 0 qB automatically reverts the last value.

Reith
Newbie
Newbie
Posts: 11
Joined: Wed Sep 26, 2012 8:17 am

Re: Caching mechanism mem overconsumption

Post by Reith » Sat Oct 20, 2012 12:25 am

I've noticed qBT seems to have a problem with memory consumption slowly creeping upwards the longer I leave it open, regardless of what it's doing.

It uses several times what I used to experience in uTorrent, which I was able to leave running for weeks at a time without running into this issue.  Here's an attachment of how it looks on my system:
Attachments
qbt.jpg

tekko

Re: Caching mechanism mem overconsumption

Post by tekko » Sat Oct 20, 2012 6:16 am

[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]

https://github.com/qbittorrent/qBittorr ... nt-9209554

For us, it would be 3.0.7.

User avatar
Dayman
Administrator
Administrator
Posts: 343
Joined: Mon Jun 18, 2012 8:12 am

Re: Caching mechanism mem overconsumption

Post by Dayman » Sat Oct 20, 2012 9:23 am

[quote="tekko"]
[quote="Dayman"]
Auto was introduced in 3.0.6 (set write cache to 0).
[/quote]

https://github.com/qbittorrent/qBittorr ... nt-9209554

For us, it would be 3.0.7.
[/quote]
Damn, too early for a report it seems. Will have to wait for 3.0.7 to be publicly available then.
Building x86_64 stuff on Windows due to Gentoo syndrome.

Jabber: [email protected]
Skype: x.dayman
Irc #[email protected]: Dayman

tekko

Re: Caching mechanism mem overconsumption

Post by tekko » Thu Nov 29, 2012 1:47 am

Could setting "sessionSettings.cache_size" be the problem? It seems to handle the read cache also.

http://www.rasterbar.com/products/libto ... anual.html

"cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).
...
use_read_cache, is set to true (default), the disk cache is also used to cache pieces read from disk. Blocks for writing pieces takes presedence."

sledgehammer_999
Administrator
Administrator
Posts: 2413
Joined: Sun Jan 23, 2011 1:17 pm

Re: Caching mechanism mem overconsumption

Post by sledgehammer_999 » Thu Nov 29, 2012 11:58 pm

From the libtorrent API documentation:
cache_size is the disk write and read cache. It is specified in units of 16 KiB blocks. Buffers that are part of a peer's send or receive buffer also count against this limit. Send and receive buffers will never be denied to be allocated, but they will cause the actual cached blocks to be flushed or evicted. If this is set to -1, the cache size is automatically set to the amount of physical RAM available in the machine divided by 8. If the amount of physical RAM cannot be determined, it's set to 1024 (= 16 MiB).

Disk buffers are allocated using a pool allocator, the number of blocks that are allocated at a time when the pool needs to grow can be specified in cache_buffer_chunk_size. This defaults to 16 blocks. Lower numbers saves memory at the expense of more heap allocations. It must be at least 1.
Emphasis mine. When the user selects "0(auto)" the cache_size variable is set to -1. I think this commit should be partially reverted.

User avatar
Dayman
Administrator
Administrator
Posts: 343
Joined: Mon Jun 18, 2012 8:12 am

Re: Caching mechanism mem overconsumption

Post by Dayman » Fri Nov 30, 2012 4:41 pm

I've reported this on qBt bugtracker: https://github.com/qbittorrent/qBittorrent/issues/258

So this is how I see this. LT uses the cache size value for write, read cache and network buffers. And what happens looks to me like this (using libastral w/o reading LT code):
  1. qBt starts; usual warm-up period
  2. after some time we hit cache limit
  3. old cache items are evicted only when we hit cache limit
  4. we stay with cache size == cache limit size forever
Well, looks like LT has no periodical GC of any sorts.
Building x86_64 stuff on Windows due to Gentoo syndrome.

Jabber: [email protected]
Skype: x.dayman
Irc #[email protected]: Dayman

User avatar
Tomaso
Veteran
Veteran
Posts: 269
Joined: Thu Dec 30, 2010 12:54 pm

Re: Caching mechanism mem overconsumption

Post by Tomaso » Sat Dec 01, 2012 10:51 am

[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]

User avatar
Dayman
Administrator
Administrator
Posts: 343
Joined: Mon Jun 18, 2012 8:12 am

Re: Caching mechanism mem overconsumption

Post by Dayman » Sat Dec 01, 2012 12:01 pm

[quote="Tomaso"]
[quote="libtorrent aio changelog"]
* improved disk cache complexity O(1) instead of O(log(n))
* add feature to allow storing disk cache blocks in an mmapped file (presumably on an SSD)
* optimized disk cache
[/quote]
[/quote]
I'll try aio HEAD then.
Doesn't build w/ bufs_size redefinition. I'll leave it at that and wait for 0.16.6 release :S
Last edited by Dayman on Sat Dec 01, 2012 12:24 pm, edited 1 time in total.
Building x86_64 stuff on Windows due to Gentoo syndrome.

Jabber: [email protected]
Skype: x.dayman
Irc #[email protected]: Dayman

User avatar
Nemo
Administrator
Administrator
Posts: 1579
Joined: Sat Jul 24, 2010 6:24 pm

Re: Caching mechanism mem overconsumption

Post by Nemo » Sat Dec 01, 2012 1:53 pm

The devs at libtorrent are trying to improve on this problem I see, thats good of course :).
Forum Rules and Guidelines

Forum Admin.
Dutch & Turkish Translator.


Image
Image

Post Reply