qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Windows specific questions, problems.
nub

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by nub »

[quote="loki"]
[quote="nub"]
So any news on this guys ? it is extremely annoying, is it going to get fixed soon or i'll have to go looking for a new torrent client ?

I am getting that "Not enough space" error, cant use this client on 1gbps connection... need to have big disk cache.
[/quote]

Given that you have one post, I would assume you are a new user. Have you tried default disk cache? What if you increase the disk cache SOME, not greater than 500MB? Have you tried the latest alpha build, and what were the results? Do you have a system that can keep up with a 1gbps connection, enough RAM, fast HDD, and see different results with, for example, utorrent?

And then the general questions you should have told us... Are you on a 64bit system? What version of Windows are you using?
[/quote]

Yes, i am a new user, after the release of version 3.1.0 this problems SEEMS to be fixed, i am not longer getting that error message so far.

PC configuration is good, no problems on that side, currently i am on 100/100 connection so i am not able to test with 1gbps but i will sometime in november when i get it back and will post if everything is good or not, going thru some ISP changes at the moment so won't be able to get my hands on 1gbps earlier than november.
Switeck

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by Switeck »

STILL sometimes crashes-to-desktop when opening Options with official 3.1.0 release...
Offset:  00168a17

Memory use exceeds 2 GB even when cache is set to 1600 MB.
There seems to resemble a big memory leak...

Even without exceeding 2 GB ram used, I get the same old error with cache set to 1500 MB:
20/10/2013 08:54:56 - An I/O error occurred, 'Torrent' paused.
20/10/2013 08:54:56 - Reason: Torrent file () error: Not enough space
sledgehammer_999
Administrator
Administrator
Posts: 2443
Joined: Sun Jan 23, 2011 1:17 pm

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by sledgehammer_999 »

Probably a leak, but I cringe on the idea on doing tests on Windows. On Linux I would just run it through valgrind. Now, I don't even know if there is an easy to use tool on Windows.
nub

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by nub »

I've opened a separate topic for this as i don't want to offtopic here.

http://qbforums.shiki.hu/index.php/topic,2325.0.html
Last edited by nub on Sat Dec 07, 2013 6:20 pm, edited 1 time in total.
loki

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by loki »

I think it only uses a single processor/core, but I could be mistaken. And wouldn't it have to be compiled/coded into a multi-processor program? (question to developer)

I would also ask, 75MB/s is very fast and could very well be the top speed for writing standard desktop HDD, what is your setup for theoretic AND (if you can) actual writing speed? I think the program I used to use is called Atto disk benchmark.
nub

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by nub »

I've opened a separate topic for this as i don't want to offtopic here.

http://qbforums.shiki.hu/index.php/topic,2325.0.html
Last edited by nub on Sat Dec 07, 2013 6:21 pm, edited 1 time in total.
Switeck

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by Switeck »

STILL crashes-to-desktop when opening Options with official 3.1.9.2 release!
Offset:  0016c5b3
...using only a 1500 MB cache.

Memory usage at 1000 MB cache: 1,316,000 KB
Memory usage at 1200 MB cache: 1,580,000 KB 264 MB
Memory usage at 1300 MB cache: 1,711,000 KB 131 MB
Memory usage at 1400 MB cache: 1,842,000 KB 131 MB
Memory usage at 1450 MB cache: 1,911,000 KB 69 MB
Memory usage at 1500 MB cache: ~1,980,000 KB CRASH!

Each 100 MB increase of the cache size increased qBT's ram memory usage by ~131 MB. While the indexing of the cache should understandably take up extra ram, the index should be <5% of the cache's size.

When I didn't get a crash-to-desktop crash (because I didn't try to open the options menu while nearing max ram usage), I get this error:
07/05/2014 14:45:28 - An I/O error occurred, 'Torrent' paused.
07/05/2014 14:45:28 - Reason: Torrent file () error: Not enough space
ciaobaby

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by ciaobaby »

qBitTorrent is a 32 bit application and WILL crash if cache usage gets near 2.0 Gib.
Switeck

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by Switeck »

qBitTorrent has a memory leak bug that causes it to reach that 32 bit app hard limit sooner than it should.

In earlier versions, the crash occurred at much lower ram usage...so there has been improvement.
ciaobaby

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by ciaobaby »

First off, under no circumstance should you need a cache greater than 256 - 320 kB.

The BitTorrent protocol moves data in 16KiB blocks, and you only need to reserve cache space for the number of blocks that are expected to be active at any point in time, 256KiB gives space for 16, 320KiB gives 20 active. Now unless you have a thousand active downloads and each download  has a hundred plus connected seeds, your client having more than twenty incoming pieces every minute of every hour of every day simply ISN'T going to happen.

Just let the client manage it's own cache, it knows far better and far quicker than you do just how much cache space is ACTUALLY NEEDED at any point in time.

To be honest if I were compiling ANY BitTorrent client for M$ Windows, cache management by the user would be disabled completely.
Switeck

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by Switeck »

This is a bug, but it's up to them whether they want to block "ridiculous" settings or not.

Spinning disk hard drives handle sequential bursts of data far better than random reads/writes.
Even SSDs have issues with high I/O rates with small writes.
So some BT clients hold downloaded data in their cache and only write it to disk as whole pieces or after 1-5 minutes inactive, whichever comes first.
Due to multi-file torrents not having consistent end-file + piece alignment, that strategy may not be advantageous.

Considering piece sizes on many torrents are 1, 2, or even 4 MB size...this can be a reason on those types of BT clients to hold the equivalent of a few pieces in the write or read cache at once.
Naturally, this depends on the speed of the internet connection, number of active upload slots, how many peers are (needlessly?) connected at once, and how many torrents are currently active.
ciaobaby

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by ciaobaby »

Considering piece sizes on many torrents are 1, 2, or even 4 MB size...this can be a reason on those types of BT clients to hold the equivalent of a few pieces in the write or read cache at once.
You are missing (or ignoring)  one important fact. BitTorrent protocol clients do NOT transfer an ENTIRE piece across the networks, they do NOT cache an entire piece in memory. The transfer and caching  is in 16k BLOCKS.
On the other hand, too large pieces (4 MB and larger) can slow down piece distribution, as a single piece will have 256 or more 16 kB "blocks", which are the actual smallest transmission units in the bittorent protocol. That will lead to rather large "transmission-in-progress" data amounts (unfinished pieces) for each client. One should consider very carefully before creating piece sizes over 1 MB even for large torrents.
http://wiki.vuze.com/w/Torrent_Piece_Size
sledgehammer_999
Administrator
Administrator
Posts: 2443
Joined: Sun Jan 23, 2011 1:17 pm

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by sledgehammer_999 »

[quote="ciaobaby"]
You are missing (or ignoring)  one important fact. BitTorrent protocol clients do NOT transfer an ENTIRE piece across the networks, they do NOT cache an entire piece in memory. The transfer and caching  is in 16k BLOCKS.[/quote]

Even if this is true, you get far better disk throughput if you cache your 16KB BLOCKS and issue one write command that covers all your cached 16KB blocks as one data chunk, rather than constantly issuing write commands of 16KB data chunks.

I imagine the same take place when seeding. It is better if you have the whole(ideally) file in memory and serving it from there rather constantly issuing read commands of 16KB chunks. Copying data from RAM is far quicker than copying it from disk. So if you have a really really fast uplink your HDD will end up throttling your uploading because it cannot keep up. So its better to cache the 16KB chunk in case someone else requests it.

Of course all this needs careful optimization eg cache expiration(you don't want to keep the data forever in memory) and depends on workload eg it is different if you seed only one file vs multiple files or famous seeds vs dead torrents....
ciaobaby

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by ciaobaby »

you get far better disk throughput if you cache your 16KB BLOCKS and issue one write command that covers all your cached 16KB blocks as one data chunk, rather than constantly issuing write commands of 16KB data chunks.
Of course, the difference is that in reality, the client probably needs nowhere near the amount of cache that users think, and setting an unnecessarily large cache leads on to the well known M$ Windows cache 'run-away' problem.

I have run and tested many BT clients, on both Windows and Linux and the only ones I have ever needed to manage the cache sizes on  is uTorrent on Windows, and there is was needed to limit it to 256k or less because leaving it at the default of 32MB tended to  degrade it's performance. More is not always 'better' despite what many YouTube videos may tell you.
Switeck

Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails

Post by Switeck »

If I have 40 active peers requesting data from my connection at once (ie: 40 active upload slots, possibly spread between multiple torrents) then each one could have one 16 KB cached chunk.
That's 640 KB right there.
But just like streaming video, that's only enough for barely-in-time results and lacks a real buffer against 1-15 second pipeline stalls.

Many disk caches even do speculative read-ahead, which doesn't work with BitTorrent clients grabbing semi-random 16 KB chunks.

Hard drives are horrible for small disk reads/writes. CrystalDiskMark is a Disk Benchmark program that can measure disk speeds in MB/second.
[2014/04/05] CrystalDiskMark 3.0.3b - Disk Benchmark
http://crystalmark.info/?lang=en
http://crystalmark.info/redirect.php?pr ... staller-en

Block Size = 4KB may not be equal to 16 KB chunk size that BitTorrent uses, however it can give you a good rule-of-thumb for just how badly a hard drive does with random reads/writes like you suggest.
Post Reply