qBT 3.0.10 >500MB cache size can trigger crash + hashfails
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
[quote="loki"]
[quote="nub"]
So any news on this guys ? it is extremely annoying, is it going to get fixed soon or i'll have to go looking for a new torrent client ?
I am getting that "Not enough space" error, cant use this client on 1gbps connection... need to have big disk cache.
[/quote]
Given that you have one post, I would assume you are a new user. Have you tried default disk cache? What if you increase the disk cache SOME, not greater than 500MB? Have you tried the latest alpha build, and what were the results? Do you have a system that can keep up with a 1gbps connection, enough RAM, fast HDD, and see different results with, for example, utorrent?
And then the general questions you should have told us... Are you on a 64bit system? What version of Windows are you using?
[/quote]
Yes, i am a new user, after the release of version 3.1.0 this problems SEEMS to be fixed, i am not longer getting that error message so far.
PC configuration is good, no problems on that side, currently i am on 100/100 connection so i am not able to test with 1gbps but i will sometime in november when i get it back and will post if everything is good or not, going thru some ISP changes at the moment so won't be able to get my hands on 1gbps earlier than november.
[quote="nub"]
So any news on this guys ? it is extremely annoying, is it going to get fixed soon or i'll have to go looking for a new torrent client ?
I am getting that "Not enough space" error, cant use this client on 1gbps connection... need to have big disk cache.
[/quote]
Given that you have one post, I would assume you are a new user. Have you tried default disk cache? What if you increase the disk cache SOME, not greater than 500MB? Have you tried the latest alpha build, and what were the results? Do you have a system that can keep up with a 1gbps connection, enough RAM, fast HDD, and see different results with, for example, utorrent?
And then the general questions you should have told us... Are you on a 64bit system? What version of Windows are you using?
[/quote]
Yes, i am a new user, after the release of version 3.1.0 this problems SEEMS to be fixed, i am not longer getting that error message so far.
PC configuration is good, no problems on that side, currently i am on 100/100 connection so i am not able to test with 1gbps but i will sometime in november when i get it back and will post if everything is good or not, going thru some ISP changes at the moment so won't be able to get my hands on 1gbps earlier than november.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
STILL sometimes crashes-to-desktop when opening Options with official 3.1.0 release...
Offset: 00168a17
Memory use exceeds 2 GB even when cache is set to 1600 MB.
There seems to resemble a big memory leak...
Even without exceeding 2 GB ram used, I get the same old error with cache set to 1500 MB:
20/10/2013 08:54:56 - An I/O error occurred, 'Torrent' paused.
20/10/2013 08:54:56 - Reason: Torrent file () error: Not enough space
Offset: 00168a17
Memory use exceeds 2 GB even when cache is set to 1600 MB.
There seems to resemble a big memory leak...
Even without exceeding 2 GB ram used, I get the same old error with cache set to 1500 MB:
20/10/2013 08:54:56 - An I/O error occurred, 'Torrent' paused.
20/10/2013 08:54:56 - Reason: Torrent file () error: Not enough space
-
- Administrator
- Posts: 2443
- Joined: Sun Jan 23, 2011 1:17 pm
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
Probably a leak, but I cringe on the idea on doing tests on Windows. On Linux I would just run it through valgrind. Now, I don't even know if there is an easy to use tool on Windows.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
I've opened a separate topic for this as i don't want to offtopic here.
http://qbforums.shiki.hu/index.php/topic,2325.0.html
http://qbforums.shiki.hu/index.php/topic,2325.0.html
Last edited by nub on Sat Dec 07, 2013 6:20 pm, edited 1 time in total.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
I think it only uses a single processor/core, but I could be mistaken. And wouldn't it have to be compiled/coded into a multi-processor program? (question to developer)
I would also ask, 75MB/s is very fast and could very well be the top speed for writing standard desktop HDD, what is your setup for theoretic AND (if you can) actual writing speed? I think the program I used to use is called Atto disk benchmark.
I would also ask, 75MB/s is very fast and could very well be the top speed for writing standard desktop HDD, what is your setup for theoretic AND (if you can) actual writing speed? I think the program I used to use is called Atto disk benchmark.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
I've opened a separate topic for this as i don't want to offtopic here.
http://qbforums.shiki.hu/index.php/topic,2325.0.html
http://qbforums.shiki.hu/index.php/topic,2325.0.html
Last edited by nub on Sat Dec 07, 2013 6:21 pm, edited 1 time in total.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
STILL crashes-to-desktop when opening Options with official 3.1.9.2 release!
Offset: 0016c5b3
...using only a 1500 MB cache.
Memory usage at 1000 MB cache: 1,316,000 KB
Memory usage at 1200 MB cache: 1,580,000 KB 264 MB
Memory usage at 1300 MB cache: 1,711,000 KB 131 MB
Memory usage at 1400 MB cache: 1,842,000 KB 131 MB
Memory usage at 1450 MB cache: 1,911,000 KB 69 MB
Memory usage at 1500 MB cache: ~1,980,000 KB CRASH!
Each 100 MB increase of the cache size increased qBT's ram memory usage by ~131 MB. While the indexing of the cache should understandably take up extra ram, the index should be <5% of the cache's size.
When I didn't get a crash-to-desktop crash (because I didn't try to open the options menu while nearing max ram usage), I get this error:
07/05/2014 14:45:28 - An I/O error occurred, 'Torrent' paused.
07/05/2014 14:45:28 - Reason: Torrent file () error: Not enough space
Offset: 0016c5b3
...using only a 1500 MB cache.
Memory usage at 1000 MB cache: 1,316,000 KB
Memory usage at 1200 MB cache: 1,580,000 KB 264 MB
Memory usage at 1300 MB cache: 1,711,000 KB 131 MB
Memory usage at 1400 MB cache: 1,842,000 KB 131 MB
Memory usage at 1450 MB cache: 1,911,000 KB 69 MB
Memory usage at 1500 MB cache: ~1,980,000 KB CRASH!
Each 100 MB increase of the cache size increased qBT's ram memory usage by ~131 MB. While the indexing of the cache should understandably take up extra ram, the index should be <5% of the cache's size.
When I didn't get a crash-to-desktop crash (because I didn't try to open the options menu while nearing max ram usage), I get this error:
07/05/2014 14:45:28 - An I/O error occurred, 'Torrent' paused.
07/05/2014 14:45:28 - Reason: Torrent file () error: Not enough space
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
qBitTorrent is a 32 bit application and WILL crash if cache usage gets near 2.0 Gib.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
qBitTorrent has a memory leak bug that causes it to reach that 32 bit app hard limit sooner than it should.
In earlier versions, the crash occurred at much lower ram usage...so there has been improvement.
In earlier versions, the crash occurred at much lower ram usage...so there has been improvement.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
First off, under no circumstance should you need a cache greater than 256 - 320 kB.
The BitTorrent protocol moves data in 16KiB blocks, and you only need to reserve cache space for the number of blocks that are expected to be active at any point in time, 256KiB gives space for 16, 320KiB gives 20 active. Now unless you have a thousand active downloads and each download has a hundred plus connected seeds, your client having more than twenty incoming pieces every minute of every hour of every day simply ISN'T going to happen.
Just let the client manage it's own cache, it knows far better and far quicker than you do just how much cache space is ACTUALLY NEEDED at any point in time.
To be honest if I were compiling ANY BitTorrent client for M$ Windows, cache management by the user would be disabled completely.
The BitTorrent protocol moves data in 16KiB blocks, and you only need to reserve cache space for the number of blocks that are expected to be active at any point in time, 256KiB gives space for 16, 320KiB gives 20 active. Now unless you have a thousand active downloads and each download has a hundred plus connected seeds, your client having more than twenty incoming pieces every minute of every hour of every day simply ISN'T going to happen.
Just let the client manage it's own cache, it knows far better and far quicker than you do just how much cache space is ACTUALLY NEEDED at any point in time.
To be honest if I were compiling ANY BitTorrent client for M$ Windows, cache management by the user would be disabled completely.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
This is a bug, but it's up to them whether they want to block "ridiculous" settings or not.
Spinning disk hard drives handle sequential bursts of data far better than random reads/writes.
Even SSDs have issues with high I/O rates with small writes.
So some BT clients hold downloaded data in their cache and only write it to disk as whole pieces or after 1-5 minutes inactive, whichever comes first.
Due to multi-file torrents not having consistent end-file + piece alignment, that strategy may not be advantageous.
Considering piece sizes on many torrents are 1, 2, or even 4 MB size...this can be a reason on those types of BT clients to hold the equivalent of a few pieces in the write or read cache at once.
Naturally, this depends on the speed of the internet connection, number of active upload slots, how many peers are (needlessly?) connected at once, and how many torrents are currently active.
Spinning disk hard drives handle sequential bursts of data far better than random reads/writes.
Even SSDs have issues with high I/O rates with small writes.
So some BT clients hold downloaded data in their cache and only write it to disk as whole pieces or after 1-5 minutes inactive, whichever comes first.
Due to multi-file torrents not having consistent end-file + piece alignment, that strategy may not be advantageous.
Considering piece sizes on many torrents are 1, 2, or even 4 MB size...this can be a reason on those types of BT clients to hold the equivalent of a few pieces in the write or read cache at once.
Naturally, this depends on the speed of the internet connection, number of active upload slots, how many peers are (needlessly?) connected at once, and how many torrents are currently active.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
You are missing (or ignoring) one important fact. BitTorrent protocol clients do NOT transfer an ENTIRE piece across the networks, they do NOT cache an entire piece in memory. The transfer and caching is in 16k BLOCKS.Considering piece sizes on many torrents are 1, 2, or even 4 MB size...this can be a reason on those types of BT clients to hold the equivalent of a few pieces in the write or read cache at once.
http://wiki.vuze.com/w/Torrent_Piece_SizeOn the other hand, too large pieces (4 MB and larger) can slow down piece distribution, as a single piece will have 256 or more 16 kB "blocks", which are the actual smallest transmission units in the bittorent protocol. That will lead to rather large "transmission-in-progress" data amounts (unfinished pieces) for each client. One should consider very carefully before creating piece sizes over 1 MB even for large torrents.
-
- Administrator
- Posts: 2443
- Joined: Sun Jan 23, 2011 1:17 pm
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
[quote="ciaobaby"]
You are missing (or ignoring) one important fact. BitTorrent protocol clients do NOT transfer an ENTIRE piece across the networks, they do NOT cache an entire piece in memory. The transfer and caching is in 16k BLOCKS.[/quote]
Even if this is true, you get far better disk throughput if you cache your 16KB BLOCKS and issue one write command that covers all your cached 16KB blocks as one data chunk, rather than constantly issuing write commands of 16KB data chunks.
I imagine the same take place when seeding. It is better if you have the whole(ideally) file in memory and serving it from there rather constantly issuing read commands of 16KB chunks. Copying data from RAM is far quicker than copying it from disk. So if you have a really really fast uplink your HDD will end up throttling your uploading because it cannot keep up. So its better to cache the 16KB chunk in case someone else requests it.
Of course all this needs careful optimization eg cache expiration(you don't want to keep the data forever in memory) and depends on workload eg it is different if you seed only one file vs multiple files or famous seeds vs dead torrents....
You are missing (or ignoring) one important fact. BitTorrent protocol clients do NOT transfer an ENTIRE piece across the networks, they do NOT cache an entire piece in memory. The transfer and caching is in 16k BLOCKS.[/quote]
Even if this is true, you get far better disk throughput if you cache your 16KB BLOCKS and issue one write command that covers all your cached 16KB blocks as one data chunk, rather than constantly issuing write commands of 16KB data chunks.
I imagine the same take place when seeding. It is better if you have the whole(ideally) file in memory and serving it from there rather constantly issuing read commands of 16KB chunks. Copying data from RAM is far quicker than copying it from disk. So if you have a really really fast uplink your HDD will end up throttling your uploading because it cannot keep up. So its better to cache the 16KB chunk in case someone else requests it.
Of course all this needs careful optimization eg cache expiration(you don't want to keep the data forever in memory) and depends on workload eg it is different if you seed only one file vs multiple files or famous seeds vs dead torrents....
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
Of course, the difference is that in reality, the client probably needs nowhere near the amount of cache that users think, and setting an unnecessarily large cache leads on to the well known M$ Windows cache 'run-away' problem.you get far better disk throughput if you cache your 16KB BLOCKS and issue one write command that covers all your cached 16KB blocks as one data chunk, rather than constantly issuing write commands of 16KB data chunks.
I have run and tested many BT clients, on both Windows and Linux and the only ones I have ever needed to manage the cache sizes on is uTorrent on Windows, and there is was needed to limit it to 256k or less because leaving it at the default of 32MB tended to degrade it's performance. More is not always 'better' despite what many YouTube videos may tell you.
Re: qBT 3.0.10 >500MB cache size can trigger crash + hashfails
If I have 40 active peers requesting data from my connection at once (ie: 40 active upload slots, possibly spread between multiple torrents) then each one could have one 16 KB cached chunk.
That's 640 KB right there.
But just like streaming video, that's only enough for barely-in-time results and lacks a real buffer against 1-15 second pipeline stalls.
Many disk caches even do speculative read-ahead, which doesn't work with BitTorrent clients grabbing semi-random 16 KB chunks.
Hard drives are horrible for small disk reads/writes. CrystalDiskMark is a Disk Benchmark program that can measure disk speeds in MB/second.
[2014/04/05] CrystalDiskMark 3.0.3b - Disk Benchmark
http://crystalmark.info/?lang=en
http://crystalmark.info/redirect.php?pr ... staller-en
Block Size = 4KB may not be equal to 16 KB chunk size that BitTorrent uses, however it can give you a good rule-of-thumb for just how badly a hard drive does with random reads/writes like you suggest.
That's 640 KB right there.
But just like streaming video, that's only enough for barely-in-time results and lacks a real buffer against 1-15 second pipeline stalls.
Many disk caches even do speculative read-ahead, which doesn't work with BitTorrent clients grabbing semi-random 16 KB chunks.
Hard drives are horrible for small disk reads/writes. CrystalDiskMark is a Disk Benchmark program that can measure disk speeds in MB/second.
[2014/04/05] CrystalDiskMark 3.0.3b - Disk Benchmark
http://crystalmark.info/?lang=en
http://crystalmark.info/redirect.php?pr ... staller-en
Block Size = 4KB may not be equal to 16 KB chunk size that BitTorrent uses, however it can give you a good rule-of-thumb for just how badly a hard drive does with random reads/writes like you suggest.