Stop Grinding my Hard Drive, please.

Discuss suggestions and ideas for the forums, site, software.
Post Reply
tekno
Newbie
Newbie
Posts: 8
Joined: Sat Sep 02, 2017 4:38 am

Stop Grinding my Hard Drive, please.

Post by tekno »

QBittorrent is a nice program. I like it. But it has the bad habit of periodically grinding my hard drive.  Yes, I do have a hard drive, not SSD.  I found this excessive hard drive seeking is caused by QBittorrent's fast resume feature.  I am not sure what the purpose of the fast resume is.  Is fast resume different than regular resume? 

The fast resume files are located in "local settings\qBittorrent\BT_backup" folder.  Most of these files are a few KB each, only 1.8 MB in total.  I found 380 files created in 21 seconds.  That's 55 milliseconds per file, due to the random seeks of a HDD.  It appears the fast resume files are created at intervals of 18 minutes.  This places a potential upper limit on the number of torrents or fast resumes QBittorent can support.  (18min*60)/0.055 sec = 19600 torrents.  If someone had that large number of torrents, then QBittorrent would spend all of its time creating fast resumes and not leave the hard drive for other uses.

Here is a solution.  The total amount of data for fast resume is small, only a few MB.  The fast resume data should be buffered in memory and then written to disk as one file instead of numerous smaller files.  This would avoid the numerous, slow random seeks on a hard drive.

Anyone else notice this issue, especially with a large number of torrents?
Switeck
Forum addict
Forum addict
Posts: 1520
Joined: Sat Jul 06, 2013 10:04 pm

Re: Stop Grinding my Hard Drive, please.

Post by Switeck »

Good find! And yes, that definitely needs to be changed. But it would probably require changes to libtorrent that qBitTorrent uses rather than qBitTorrent itself.

On top of the issue you found, there's also the issue of torrent file fragmentation especially for large torrents...which can mean a HDD has even less time to spend reading/writing resumes:
1.Misaligned multi-file torrents cause slow speeds - index.php/topic,2627.msg12725.html#msg12725
2.Neither preallocate files nor sparse files works well - index.php/topic,3787.0.html  and    index.php/topic,3787.msg19159.html#msg19159    and http://forum.deluge-torrent.org/viewtopic.php?t=27725
3.sparse file handling causes horrific file fragmentation - index.php/topic,3743.msg19531.html#msg19531    and    https://github.com/qbittorrent/qBittorr ... /4124  qBittorrent performs horribly wrong disk I/O
tekno
Newbie
Newbie
Posts: 8
Joined: Sat Sep 02, 2017 4:38 am

Re: Stop Grinding my Hard Drive, please.

Post by tekno »

[quote="Switeck"]
Good find! And yes, that definitely needs to be changed. But it would probably require changes to libtorrent that qBitTorrent uses rather than qBitTorrent itself.
[/quote]

Thanks for the links.  Informative reading.  I have had a suspicion that QBittorrent causes the downloaded files to be fragmented due to downloading peaces in a random fashion.  That might be a different topic.

I did some research on QBittorrent and Libtorrent and downloaded the source codes.  Indeed, the fast resume is a feature of libtorrent.  But, I wonder if QBittorrent abuses the fast resume feature.  From the documentation for libtorrent, it looks like fast resume is mostly about saving the state of incomplete torrents.  Is that true?  Almost all of my torrents are in the "complete/seeding" state, but QBittorrent still periodically saves fast resume data for all of them.  Is this correct?  Should QBittorrent frequently save fast resume data for completed/seeding torrents?  I tried to read the code, but I am not a C++ programmer.  I have the sense that Libtorrent generates the fast resume data and QBittorrent saves the fastresume files, but I'm not sure.  Again, I'm not a C++ programmer.  I'm thinking that QBittorrent can be changed to buffer all fast resume data and save in one file.  Or, else the fast resume data should only be generated on active/incomplete/downloading torrents.

Here are a few quotes from the libtorrent documentation:


http://www.libtorrent.org/features.html
fast resume support, a way to avoid the costly piece check at the start of a resumed torrent. Saves the storage state, piece_picker state as well as all local peers in a fast-resume file.


http://www.rasterbar.com/products/libto ... esume-data
It is still a good idea to save resume data periodically during download as well as when closing down.

When saving resume data periodically, it makes sense to skip any torrent which hasn't downloaded anything since the last time.

It is typically a good idea to save resume data whenever a torrent is completed or paused. In those cases you don't need to pause the torrent or the session, since the torrent will do no more writing to its files. If you save resume data for torrents when they are paused, you can accelerate the shutdown process by not saving resume data again for paused torrents. Completed torrents should have their resume data saved when they complete and on exit, since their statistics might be updated.

The fast resume mechanism is a way to remember which pieces are downloaded and where they are put between sessions. You can generate fast resume data by calling save_resume_data() on torrent_handle. You can then save this data to disk and use it when resuming the torrent. libtorrent will not check the piece hashes then, and rely on the information given in the fast-resume data. The fast-resume data also contains information about which blocks, in the unfinished pieces, were downloaded, so it will not have to start from scratch on the partially downloaded pieces.
Switeck
Forum addict
Forum addict
Posts: 1520
Joined: Sat Jul 06, 2013 10:04 pm

Re: Stop Grinding my Hard Drive, please.

Post by Switeck »

Probably a core issue of libtorrent, so qBT devs can't fix it. :(
User avatar
dragons4life
Veteran
Veteran
Posts: 120
Joined: Sat Dec 20, 2014 5:16 am

Re: Stop Grinding my Hard Drive, please.

Post by dragons4life »

Almost certain this happens with every other client, and also depends on your HDD.
User avatar
Peter
Administrator
Administrator
Posts: 1949
Joined: Wed Jul 07, 2010 6:14 pm

Re: Stop Grinding my Hard Drive, please.

Post by Peter »

- The write should not cause serious problems.
- You may just have a bad or slow hard drive. Get a WD Black at least.
- It helps tremendously if you have your OS on one drive (SSD is better but a HDD also works great), and you have a separate drive where you download to (and load programs and whatnot from.)

I also used just two WD Black HDD drives in RAID0 for 5+ years. Most motherboards support it, and it's crazy great.
- qBittorrent team - server and forum administrator.
- Hungarian translation reviewer/moderator (+ translator).

Join the official qBittorrent Discord!
https://discord.gg/ma66Vv4
User avatar
dragons4life
Veteran
Veteran
Posts: 120
Joined: Sat Dec 20, 2014 5:16 am

Re: Stop Grinding my Hard Drive, please.

Post by dragons4life »

[quote="Peter"]
- The write should not cause serious problems.
- You may just have a bad or slow hard drive. Get a WD Black at least.
- It helps tremendously if you have your OS on one drive (SSD is better but a HDD also works great), and you have a separate drive where you download to (and load programs and whatnot from.)

I also used just two WD Black HDD drives in RAID0 for 5+ years. Most motherboards support it, and it's crazy great.
[/quote]
I am pretty certain mines has to be one of the shittiest drives in the entire forum, can hear clicking sounds just from uploading (less than 1mb speed).
User avatar
Peter
Administrator
Administrator
Posts: 1949
Joined: Wed Jul 07, 2010 6:14 pm

Re: Stop Grinding my Hard Drive, please.

Post by Peter »

Ouch.
Should we put up a gofundme banner on the forums to fund a new hard drive?  ;D

Would send you one if you lived in the same country.
- qBittorrent team - server and forum administrator.
- Hungarian translation reviewer/moderator (+ translator).

Join the official qBittorrent Discord!
https://discord.gg/ma66Vv4
User avatar
dragons4life
Veteran
Veteran
Posts: 120
Joined: Sat Dec 20, 2014 5:16 am

Re: Stop Grinding my Hard Drive, please.

Post by dragons4life »

[quote="Peter"]
Ouch.
Should we put up a gofundme banner on the forums to fund a new hard drive?  ;D

Would send you one if you lived in the same country.
[/quote]
lol, I just gotta find a deal and motivation to buy it
User avatar
Peter
Administrator
Administrator
Posts: 1949
Joined: Wed Jul 07, 2010 6:14 pm

Re: Stop Grinding my Hard Drive, please.

Post by Peter »

I had this cool fella who was really into archiving. He bought drives by the dozen. So I purchased his old drives all the time to have replacements. But yeah, he switched jobs and the flood of drives stopped. I am thinking about buying a big 4TB drive or something and buying a Backblaze subscription, so basically I would have a big cloud drive that is backed up all the time.

Just a thought...
- qBittorrent team - server and forum administrator.
- Hungarian translation reviewer/moderator (+ translator).

Join the official qBittorrent Discord!
https://discord.gg/ma66Vv4
OdinWillsIt
Newbie
Newbie
Posts: 8
Joined: Mon Sep 18, 2017 3:51 pm

Re: Stop Grinding my Hard Drive, please.

Post by OdinWillsIt »

[quote="Peter"]
- The write should not cause serious problems.
- You may just have a bad or slow hard drive. Get a WD Black at least.
- It helps tremendously if you have your OS on one drive (SSD is better but a HDD also works great), and you have a separate drive where you download to (and load programs and whatnot from.)

I also used just two WD Black HDD drives in RAID0 for 5+ years. Most motherboards support it, and it's crazy great.
[/quote]

Yeah blacks are great have 3 myself so are reds but much slower


Blacks are designed for performace usually I get around 200mbps read/write minimum 150
Have 2x 1tb
1x 5tb
blacks

couple of Red 8tbs
Post Reply