Memory consumption over time

Windows specific questions, problems.
pcguy

Memory consumption over time

Post by pcguy »

I've seen other posts in the past regarding memory usage in qbittorrent, but most have been claimed to have been fixed.  I'm running the latest at this time (3.1.9) on Win7 64 bit.  I absolutely love the client!

What I'm seeing is massive memory usage that isn't reported in Task Manager or Windows Resource Monitor.  Seeding just 5 torrents (between 10-30GB each) consumed over 1GB of memory after a few hours.  Killing the qbittorrent client immediately frees the memory.

I am running over a VPN and have the Network Interface bound to Windows TAP interface.  Disk write was set to auto, I changed it to 16MiB, no difference.  Disk cache expiry is set for 60s.  Global max connection 400, max connections per torrent 100, global upload slots 8, max upload slots per torrent 4.  No proxy.  Speed set to unlimited, enable bandwidth management (uTP), Apply rate limit to uTP connections checked, and apply rate limit to transport overhead un-checked.  uPnP/NAT-PMP un-checked.  Enable DHT checked, enable Peer Exchange checked, enable local peer discoverty checked.  Enable anonymous mode un-checked.  Torrent queuing un-checked.

What can I do to help diagnose this issue further?
pcguy

Re: Memory consumption over time

Post by pcguy »

For the heck of it, upgraded to the 3.2 alpha.  Ran the same 5 torrents (seeding only) for 8 hours.  Almost 3GB of data was consumed by qBittorrent.  Killed the app, and memory usage instantly went from 6.25GB to 3.27GB.

Is it just me seeing this behavior?
ciaobaby

Re: Memory consumption over time

Post by ciaobaby »

Seems like that's the case.

I have 26 loaded jobs (all seeding currently), and there's usually 7 active at any particular time, qBT is running 24/7 and "Working Set" memory usage for qBT is under 100MiB (32bit Win 7)
massive memory usage that isn't reported in Task Manager or Windows Resource Monitor.
OK, so where or what is it being reported by, and what does is say is using that memory?
ciaobaby

Re: Memory consumption over time

Post by ciaobaby »

Oh! And do your active jobs contain several 'files' by any chance?
pcguy

Re: Memory consumption over time

Post by pcguy »

[quote="ciaobaby"]OK, so where or what is it being reported by, and what does is say is using that memory?[/quote]

Windows Task Manager, Performance Tab.  Note that looking in the Processes tab, it does not show qBittorrent using the memory, but closing the application immediately reduced the memory footprint by gigs (as shown in the attached graph).
Attachments
taskmgr.png
pcguy

Re: Memory consumption over time

Post by pcguy »

[quote="ciaobaby"]Oh! And do your active jobs contain several 'files' by any chance?[/quote]

One of them has 14 files.  The rest are 2-3 files each.
ciaobaby

Re: Memory consumption over time

Post by ciaobaby »

it does not show qBittorrent using the memory, but closing the application immediately reduced the memory footprint by gigs (as shown in the attached graph).
So by deductive reasoning the problem must be qBT.

Okay, the problem is NOT qBT, uTorrent, BitTorrent, Vuze, etc. etc. [choose your own Windows BitTorrent client]. The issue is the Windows pathetically poor caching system which holds file handles open and reserves (caches) memory to them on a 'just in case' basis, and BitTorrent clients which will be referencing many files (or so Window's 'thinks') are simply the catalyst for exposing Window's shortcomings.

My system doesn't exhibit this behaviour with BT clients because the jobs I run are all single file, such as  Linux distro ISOs, Open Source application installers etc, I do not download multi-file torrents such as TV Shows, commercial music downloads etc. so Windows does not need to reserve memory just in case I might want to open one or more of them. If you have a large music collection vinyl, CDs etc. converted to, or purchased as  MP3, FLAC etc. you WILL see the same behaviour caused by Winamp, Songbird, FooBar 200, iTunes, WMP  etc. where they have a 'library' of files that might be opened, so Windows caches them 'just in case'.
Last edited by ciaobaby on Fri Mar 07, 2014 3:04 pm, edited 1 time in total.
pcguy

Re: Memory consumption over time

Post by pcguy »

Thanks for the reply, but I'm not convinced.  I used to use uTorrent, and it did not exhaust all system memory within a 24 hour period like qBittorrent does.  Eventually qBittorrent will consume everything available - it took me about 3 minutes to locally log onto my system (disk swapping) before I knew this was an issue.  Admittedly, this *was* an issue with specific versions of libtorrent (which qBittorrent uses), but it was marked as fixed as of the version being used currently.

I'm only seeding 5 torrents with a total under 30 files.  Is that considered excessive?  Or is it total size of files (one file that is 200gig is the same as 200 1gig files)?

I'll do more experiments (using Vuze) to see if it exhibits the same behavior (last time I checked, it didn't).

Thanks...
Last edited by pcguy on Fri Mar 07, 2014 3:15 pm, edited 1 time in total.
ciaobaby

Re: Memory consumption over time

Post by ciaobaby »

Is that considered excessive?  Or is it total size of files (one file that is 200gig is the same as 200 1gig files)?
No and no.

It depends on how many 'files' are being processed by the client for pieces that 'active', i.e. being uploaded/downloaded by the client at any particular point

In an over simplified way it means that:
One 'file' with fifty active pieces only reserves space for one file,

Fifty 'files' with only one piece active for each reserves space for fifty files.
pcguy

Re: Memory consumption over time

Post by pcguy »

Thanks.  So my seeding is not excessive, but qBittorrent is causing Windows to consume all available memory.  Ugh.

I'm creating an environment to test Vuze (and perhaps uTorrent).  Same files, just seeding, and will measure consumed memory over time.
sledgehammer_999
Administrator
Administrator
Posts: 2443
Joined: Sun Jan 23, 2011 1:17 pm

Re: Memory consumption over time

Post by sledgehammer_999 »

We have gone back and forth with this problem in the past. IIRC libtorrent has 2 modes of caching things. One is to leave it to the OS and the other is to handle it itself. In the past we had the first one, then the second one and now the first one again. I don't remember why we change that each time. I think the best option here is to make it configurable and let the user decide which mode to use.
(Unless the reason of switching from manual caching to OS caching is now obsolete).
User avatar
Nemo
Administrator
Administrator
Posts: 1730
Joined: Sat Jul 24, 2010 6:24 pm

Re: Memory consumption over time

Post by Nemo »

qBittorrent is running now for 2,5 days straight with 55 torrent on seeding status, memory usage shows 31MB. Performance -> Memory is also right. As far I know if the process shows for example like mine 31MB (qBittorrent.exe) then ''Memory'' can't show more than that. Thats impossible (..or am I wrong?). But still, its indeed very high usage if it really is qBittorrent.
ciaobaby

Re: Memory consumption over time

Post by ciaobaby »

Task Manager is only a rough tool when it comes to seeing what is using resources. For a better idea of what is using memory, run Windows Resource Monitor

Start -> Run -> perfmon.exe /res

Select qbittorrent.exe in the Image column and it will filter the resource list to only what the qBT is using and what file handles are open.

System will show what Windows' is holding and using.
pcguy

Re: Memory consumption over time

Post by pcguy »

I tested the same 5 torrents with Vuze and..  memory consumption over time also went through the roof.  In about 2 hours, more than 7GB used.

I found a nifty Microsoft tool called RAMMap (http://technet.microsoft.com/en-us/sysi ... s/ff700229).  I installed that and ran selected "Empty Working Sets".  With Vuze still running, instant drop to 3.5GB used.  I then selected "Empty Standby List", and dropped it further to 2.8GB.

That said, leaving it to Windows to cache files isn't ideal (at least for some cases, like seeding large files).  Windows apparently is not smart enough to know how much memory is available.  And unfortunately, the RAMMap tool isn't scriptable.

It would be cool if there was a way for the app (like qBittorrent) to configure the *OS's* disk caching size, at least in the context of the application.  I have no idea if that is even possible in Windows.
Last edited by pcguy on Sun Mar 09, 2014 4:19 pm, edited 1 time in total.
Dayman

Re: Memory consumption over time

Post by Dayman »

Free memory = wasted memory :)
Attachments
FMWM.png
Post Reply