Few small questions about qbt-windows

Windows specific questions, problems.
internetaddict24

Few small questions about qbt-windows

Post by internetaddict24 »

Hello everyone, i just signed up here so i'm still getting to know everything around. I'd like to ask you a few questions about qBittorrent, using the latest version right now (3.3.0):

-A rather dumb question first of all: if i haven't set any download/upload limit, then will the "Apply rate limit to transport overhead" option become useless? Also, if i turned uTP off, the "Apply rate limit to uTP protocol" option will also do nothing? Are they linked to the main rate limit options, or they are doing something else and i should keep them on just like i did before?

-Even though i'm mostly downloading private torrents, i'm also sometimes downloading public ones. Should i leave DHT, PeX and LSD on? qBittorrent says it disables them ("This torrent is private" in the status message), but i had some very rare occasions where the tracker under-appreciated the upload, for example, if i uploaded 60 gb in qBittorrent the tracker only showed about 55 gb; basically i was uploading to public peers. Tried the same thing, with the same torrent, but with DHT, PeX and LSD off, and the tracker counted the upload perfectly. Does that mean they don't turn themselves off when detecting a private torrent? All the other options, like "Exchange trackers with other peers" are disabled.

-My internet connection is pretty fast: 1000 mbps down, 200 mbps up. How can i explain that, when i download at 110 MB/s the hard drive is barely loaded, but when i upload at 25 MB/s (so just a quarter of that speed) the hard drive is in constant overload and the cache fills up in a matter of minutes => qBittorrent crashes? If i limit the cache size to 1024 MB i don't have any crashes, but the speed starts variating pretty badly. It's pretty much the same number of peers for both download and upload jobs (i have limited the upload slots to 50, so that's not a ton of them stressing the hard drive), and the hard drive is a pretty fast model (1TB, single platter), i realy don't understant what's going on... Is the upload (basically the reads) not cached by Windows? :-\

Thanks in advance! :)
ciaobaby

Re: Few small questions about qbt-windows

Post by ciaobaby »

  if i haven't set any download/upload limit, then will the "Apply rate limit to transport overhead" option become useless?
Yes, the two options are mutually exclusive
Also, if i turned uTP off, the "Apply rate limit to uTP protocol" option will also do nothing?
Yes
but i had some very rare occasions where the tracker under-appreciated the upload, for example, if i uploaded 60 gb in qBittorrent the tracker only showed about 55 gb; basically i was uploading to public peers.
, i realy don't understant what's going on... Is the upload (basically the reads) not cached by Windows?
Windows crappy cache system is the problem, just leave the cache at "auto" your client will never need anything like the amount of cache reserved memory that you think it does.  For most use cases reserving 512MiB is FAR, FAR too much.
Switeck

Re: Few small questions about qbt-windows

Post by Switeck »

[quote="internetaddict24"]if i uploaded 60 gb in qBittorrent the tracker only showed about 55 gb; basically i was uploading to public peers. Tried the same thing, with the same torrent, but with DHT, PeX and LSD off, and the tracker counted the upload perfectly. Does that mean they don't turn themselves off when detecting a private torrent?[/quote]Someone probably incorrectly created those torrents as public torrents, so DHT, LSD, and PEX still worked.

I recommend leaving DHT (a whole separate network) disabled if you overwhelmingly use private trackers.

LSD can also be disabled if there's no chance of other peers/seeds being found locally either on your LAN or in the same neighborhood on the same ISP branch. Misidentified "local" seeds or peers can cause minor Tit-For-Tat issues, although it only really matters if upload is low.

PEX is best left on, it will only work with public torrents anyway. It doesn't make additional connections like DHT and LSD does and is good at finding other seeds/peers to connect to *IF* you are already connected to at least 1 seed or peer.
internetaddict24

Re: Few small questions about qbt-windows

Post by internetaddict24 »

Okay, thanks everyone!

I'll only disable DHT and leave the rest enabled, hopefully i won't be connecting to public peers anymore. One of my private trackers actually recommends leaving LSD on, possibly it has an effect with how the home networks in Romania (where i live) are placed: one main hub for each apartment building (10.x.x.x) with a PPPoE connection at the end. I noticed one time i was connected to a peer that had the same ip class and a very similar ip address as mine, that meant the peer was in the same apartment building/network hub as me? Then LSD actually has a purpose...

About the cache issue, when i had the cache size set at 1500 MiB the upload speed was staying at a constant 25 MB/s (... until qBittorrent crashed). With 1024 MiB everything is stable as a rock, but the speed starts variating between 18 ~ 25 MB/s and the HDD is under constant 100% usage (makes sense, lesser information to store in a smaller cache). My question here is: so this is most likely a hardware cause, i'm using a traditional HDD and it can't handle the load... seeding might be a more stressing task than downloading for a HDD, there are more seek tasks and the HDD can only do about 200 operations a second. Will a RAID (0) setup help? I don't have the money to spend on a 500 GB SSD  :P
Switeck

Re: Few small questions about qbt-windows

Post by Switeck »

LSD will not work with private torrents. (DHT, LSD, and PEX are all disabled there.)

If you're downloading to a solid-state drive (SSD), make sure you're not using sparse files as it's VERY hard on SSDs:
index.php/topic,2627.msg12725.html#msg12725

Storing torrents across multiple HDDs (or SSDs) not in a RAID is quicker so long as none of the drives are individually under extremely heavy loads.

Scanning the drive/s for errors and Defragging the HDDs may speed up seeding slightly. (I only recommend defragging about once a week at most.)
Not using NTFS on the drives or using preallocate for the torrent files should decrease the necessity and frequency of running a defrag.

The cache crash bug is known about and was being worked on...by limiting qBitTorrent to smaller cache size that hopefully doesn't trigger a crash:
https://github.com/qbittorrent/qBittorrent/issues/4028

qBitTorrent (and Deluge which also uses libtorrent) uses its cache poorly, and the cache is wasteful of ram memory -- using roughly 30% more ram than the cache size suggests, if the cache is full.
internetaddict24

Re: Few small questions about qbt-windows

Post by internetaddict24 »

[quote="Switeck"]Scanning the drive/s for errors and Defragging the HDDs may speed up seeding slightly. (I only recommend defragging about once a week at most.)
Not using NTFS on the drives or using preallocate for the torrent files should decrease the necessity and frequency of running a defrag.[/quote]

Preallocating might just help me in this case i think: if the torrent is stored sequentially then uploading (reading) will require much lesser seek tasks -> the HDD won't overload. At least in theory... The only problem is that if i download a very large torrent (50-100 GB) it will take a while to preallocate all of that space, and the torrent will pretty much stay paused 1 or 2 minutes before it begins.

Again, i can download at over 110 MB/s without a problem, it's only the upload that makes all these issues  :-\

[quote="Switeck"]The cache crash bug is known about and was being worked on...by limiting qBitTorrent to smaller cache size that hopefully doesn't trigger a crash:
https://github.com/qbittorrent/qBittorrent/issues/4028

qBitTorrent (and Deluge which also uses libtorrent) uses its cache poorly, and the cache is wasteful of ram memory -- using roughly 30% more ram than the cache size suggests, if the cache is full.
[/quote]

So now i know why only ~700 MB of cache fills up half of my RAM, ~3-4 GB...

I'll just leave the cache at 1024 MB, i've been seeding throught a few nights and there were no crashes whatsoever. Hopefully this issue will be fixed in the future...
ciaobaby

Re: Few small questions about qbt-windows

Post by ciaobaby »

Preallocating might just help me in this case
pre-allocation is not what you think it is. It does not allocate the required disc space, that is already done when the task is loaded, Pre-allocate merely fills the already allocated space with zero's (0).
I'll just leave the cache at 1024 MB,
Noooooo that is probably about four times the cache size you actually 'need'.

Check using View -> Statistics to see how much cache you are actually using.
Switeck

Re: Few small questions about qbt-windows

Post by Switeck »

Preallocation may not be needed on any HDD not using NTFS. In my tests, there was only 0-5 file fragments on a 14 file 1.6 GB torrent when I wasn't using preallocation on a ramdrive using FAT32 with ~4 KB cluster size. There might be a problem if you're downloading 3+ torrents at once though, because file pieces could end up alternating between torrents.

SSDs will not benefit from preallocation, so it's best to not download directly to them.
internetaddict24 wrote:So now i know why only ~700 MB of cache fills up half of my RAM, ~3-4 GB...
That's probably windows OS cache going nuts trying to hold onto every file it accesses. There's an advanced setting in qBT to disable windows OS caching of its files.
I'll just leave the cache at 1024 MB, i've been seeding throught a few nights and there were no crashes whatsoever. Hopefully this issue will be fixed in the future...
Raise the cache duration time to its maximum -- 600 seconds (10 minutes). Otherwise, it's not very useful while seeding. Oldest unused cache entries are cleared out first.
internetaddict24

Re: Few small questions about qbt-windows

Post by internetaddict24 »

[quote="ciaobaby"]pre-allocation is not what you think it is. It does not allocate the required disc space, that is already done when the task is loaded, Pre-allocate merely fills the already allocated space with zero's (0).[/quote]

Yeah, i confused that with sequential downloading.

[quote="ciaobaby"]Noooooo that is probably about four times the cache size you actually 'need'.

Check using View -> Statistics to see how much cache you are actually using.
[/quote]

800-900 MiB when downloading, and it can go all the way up to 1.5 GiB when uploading :P

[quote="Switeck"]That's probably windows OS cache going nuts trying to hold onto every file it accesses. There's an advanced setting in qBT to disable windows OS caching of its files.

Raise the cache duration time to its maximum -- 600 seconds (10 minutes). Otherwise, it's not very useful while seeding. Oldest unused cache entries are cleared out first.
[/quote]

Hmm, i'll try these out. Hopefully there won't be any significant performance loss.

If the ram consumption for qbittorrent will go down then i will be able to set the cache size to 1.5 GiB and not get any crashes. Atleast i hope so ;)
ciaobaby

Re: Few small questions about qbt-windows

Post by ciaobaby »

Is that with the cache size on 'auto'?
internetaddict24

Re: Few small questions about qbt-windows

Post by internetaddict24 »

[quote="ciaobaby"]Is that with the cache size on 'auto'?[/quote]

Yeah, it still is  :(

And disabling the OS cache does significantly reduce the amount of RAM consumed by qBittorrent, but also cuts the download speeds at ~ 80 MiB/s and the upload at ~ 14 MiB/s.

I might reconsider buying another identical HDD and put them in RAID 0, but i'm not sure if that's a good idea... I've only seen RAID 0 setups on video editing workstations, and torrenting doesn't only require fast sequential speeds, it also requires fast random accesses... But on the other hand, double the sequential speed might help with the overload caused by the crappy acces/seek times on traditional HDD's. What do you think, does it worth it?
ciaobaby

Re: Few small questions about qbt-windows

Post by ciaobaby »

And disabling the OS cache does significantly reduce the amount of RAM consumed by qBittorrent
That is because it is actually Windows using the cache for files that it 'thinks' qbittorrent is 'using', and getting it wrong.


Test with various cache sizes in the 128MiB to 512MiB range
torrenting doesn't only require fast sequential speeds, it also requires fast random accesses.
The bittorrent protocol doesn't work as you seem to think it does, which is also the reason that Windows makes such a dogs breakfast of caching with BT clients

BitTorrent clients do not really need sequential read access as they do not read entire files from disc or write 'files' to disc, fragmentation is not a major issue simply because bittorrent clients only work with file 'fragments' in any case, and the 'fragment' can be anywhere in the file structure.
internetaddict24

Re: Few small questions about qbt-windows

Post by internetaddict24 »

[quote="ciaobaby"]The bittorrent protocol doesn't work as you seem to think it does, which is also the reason that Windows makes such a dogs breakfast of caching with BT clients

BitTorrent clients do not really need sequential read access as they do not read entire files from disc or write 'files' to disc, fragmentation is not a major issue simply because bittorrent clients only work with file 'fragments' in any case, and the 'fragment' can be anywhere in the file structure.[/quote]

I was referring to sequential speeds... you know, the maximum brute speed that a HDD can acheive.

That's what i was trying to ask, if torrents are stored in fragments/pieces, then will a RAID setup still help or it will do absolutely nothing and the HDD's will still overload?

Tried reducing the cache size to various values... anything under 1000-1200 MiB will severely cut my upload speeds, it won't even reach 6-7 MiB/s. But if i download, i can even set 64 MiB of cache and the speed will still stay at a constant 110 MiB/s. That's the weird thing...

P.S. I reinstalled windows today to get a clean installation of windows 10 TH2. Installed all drivers and qBittorrent from scratch. Still the same behaviour, downloading is flawless, uploading overloads the HDD.

P.P.S. Should i make a video recording of the behaviour, just for the sake of it?
ciaobaby

Re: Few small questions about qbt-windows

Post by ciaobaby »

But if i download, i can even set 64 MiB of cache and the speed will still stay at a constant 110 MiB/s.
Not so weird, it suggests that you need to adjust your connections / upload slot ratios and your upload rate limiting to let the client run more efficiently.
internetaddict24

Re: Few small questions about qbt-windows

Post by internetaddict24 »

[quote="ciaobaby"]Not so weird, it suggests that you need to adjust your connections / upload slot ratios and your upload rate limiting to let the client run more efficiently.[/quote]

I tried limiting the connections per torrent at ~100 and the upload slots at ~15-20. Still didn't help, cache still fills up and the HDD still overloads a little bit:

Image

But again, when downloading the cache is barely even used and the HDD doesn't even break a sweat. And it's 6 times the speed and almost 3 times the amount of connections compared to the upload, that's what i found strange, exactly the opposite should happen...

Image
Last edited by internetaddict24 on Tue Dec 08, 2015 7:36 pm, edited 1 time in total.
Post Reply