Upload speed drops

[qbi]

Is that normal? Every ~18 minutes upload speed drops and immediately reaches the normal level. That happens when OS cache is enabled. When OS cache is disabled rarely happens but still happens. Also when the drops happens there is a not typical hdd noise. I tried various combinations of large/small cache + short/long cache expiry but no results. With utorrent 2.2.1 there is no such behavior.

[ciaobaby]

Every ~18 minutes upload speed drops and immediately reaches the normal level.

It might be, but how can we tell?

Input!  .... .... Need Input!

Also when the drops happens there is a not typical hdd noise.

And what 'noise' is there instead?

[qbi]

Every ~18 minutes upload speed drops and immediately reaches the normal level.

It might be, but how can we tell?

Input!  .... .... Need Input!

6 hours Look how regularly are the drops.
30 minutes When the drop happens the speed graphic looks like "1". After some time happens another drop - the first drop in the graphic is already on place "2". Every next drop happens when the previous drop is in point "2". When the drops starts I'll make screenshot on 30 minutes graphic.
When drops happened I checked the "Statistics" and "Read cache overload" goes up to ~50%. I'm only browsing meantime. The hdd read activity goes up to 50%. Also my torrents are on hdds only for torrents. I have ssd for OS.

Also when the drops happens there is a not typical hdd noise.

And what 'noise' is there instead?

When you read from hdd the noise is constant without much change in the noise level. When drop happens the initial noise is very aggressive/high and then lowering.

P.S. I'm using qBittorrent 3.3.4/Libtorrent 1.0.9.0/Boost 1.60.0. With 3.3.3 the situation was the same.
30 minutes screenshot and6 hours

[ciaobaby]

When drop happens the initial noise is very aggressive/high and then lowering.

If that was happening with a drive of mine, I would be ordering a brand new one pretty sharpish.

"Read cache overload" goes up to ~50%. I'm only browsing meantime

The two are not necessarily connected because there are many applications and services running on your machine that could interrupt or 'steal' network capacity

With 3.3.3 the situation was the same.

With what libtorrent version?
because it is libtorrent that handles all the I/O

[qbi]

When drop happens the initial noise is very aggressive/high and then lowering.

If that was happening with a drive of mine, I would be ordering a brand new one pretty sharpish.

The hdd smart is excellent. That behavior starts when I changed utorrent with qBittorrent.

"Read cache overload" goes up to ~50%. I'm only browsing meantime

The two are not necessarily connected because there are many applications and services running on your machine that could interrupt or 'steal' network capacity

Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?

With 3.3.3 the situation was the same.

With what libtorrent version?
because it is libtorrent that handles all the I/O

With default libtorrent for 3.3.3 and with 1.0.9.

P.S. I'm using 5 hdds only for torrenting without RAID.

[KitKat]

[quote="qbi"]
Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?
[/quote]
OS cache in windows is pretty bad, windows likes to preload stuff which leads to bloat.
I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.
As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.

/e: Could also always be simple HDD failure as ciao has mentioned.
If my HDD was making that kind of noise i'd be looking into replacing it asap.
SMART isnt considered reliable in business environments btw.

[qbi]

I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.

I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.

As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.

How OS cache affects packet loss, router problems and peer disconnecting?

As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.

When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.

[KitKat]

[quote="qbi"]
[quote="KitKat"]
I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.[/quote]

I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.
[/quote]
I appear to have skipped over that, sorry.
Could you try disabling OS cache entirely and seeing if qbittorents management is better than window's in your case?
The option is found under tools > options > advanced and is called Enable OS Caching, un-tick it and restart qbittorent.

[quote="qbi"]
[quote="KitKat"]
As for reasons why your upload could periodically drop, could be simple packet loss, peers disconnecting from you as theres a better seeder, or your modem/router choking.
[/quote]
How OS cache affects packet loss, router problems and peer disconnecting?
[/quote]
OS cache doesnt affect packet loss or router problems.
It affects hardware read/write which COULD be a problem if you're doing something intensive, i never mentioned those things in that context.
[quote="qbi"]
[quote="KitKat"]
As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.[/quote]

When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.
[/quote]
Browsing would involve sending TCP Handshakes which could cause Quality Of Service (QOS) router rules to drop or delay the torrent upload traffic to ensure your connection succeeds. If you're using 100% of your upload in qbittorent then QOS packet delaying could (and likely will) cause qbittorent upload drops in statistics like that, if you're not using all of your upload in qbittorent this is most likely not the cause.

For the record watching a movie can be harddrive intensive depending on the video codec and video size.
Its cache intensive at the very least.

/e:
Check MMC's Log for Read/Write Interrupts when qbittorent shows the upload drop, if they exist then whatever the process causing them is at fault.

[quote="ciaobaby"]
With what libtorrent version?
[/quote]
Not to be an asshat but the vast majority of windows users will be using the "release" builds as compiling this(most things really) under windows is a fairly tedious procedure until you have it setup..
So this question answers itself alot of the time :X

[qbi]

I'd guess you have a large default cache size set in qbittorent, try lowering it or just not using OS cache and allowing qbittorent to manage itself.

I wrote in the first post that I have tried small and large (auto, from 20MB to 2GB) cache size + long and short (from 15s to 600s) cache expiry.

I appear to have skipped over that, sorry.
Could you try disabling OS cache entirely and seeing if qbittorents management is better than window's in your case?
The option is found under tools > options > advanced and is called Enable OS Caching, un-tick it and restart qbittorent.

I mentioned in the first post that when I disable OS cache the drops happens very rarely.

As you've said it only happens when you're browsing the internet its possible you have QOS rules setup in your router to prioritise port 80/8080/443/tcp traffic and its doing its job by choking your torrent upload to perform a handshake/lookup for a fraction of a second causing qbittorent to report a drop.

When I wrote that I'm only browsing I meant that I'm doing no intensive things like copy/write on hdd or downloading. The drops happens also when I'm watching movie.

Browsing would involve sending TCP Handshakes which could cause Quality Of Service (QOS) router rules to drop or delay the torrent upload traffic to ensure your connection succeeds. If you're using 100% of your upload in qbittorent then QOS packet delaying could (and likely will) cause qbittorent upload drops in statistics like that, if you're not using all of your upload in qbittorent this is most likely not the cause.

I capped my upload speed to 90%. I have problems when global connections are > 140 (slow browsing but upload speed is fine) so I limited global connections to 130. Actually in utorrent my global connections was ~200 and I have problems with browsing but never with upload drops in the torrent client. I don't enable any QOS in the router or in OS. You can see that now the global connections are less than the limit but drops happens. Also you can see that the drops are huge - from 5-6MB/s to < 1MB/s - browsing can't cause so huge drops.

For the record watching a movie can be harddrive intensive depending on the video codec and video size.
Its cache intensive at the very least.

Also when I'm watching movie which is on other hdd there is drop and that noise. Actually I don't want disable OS cache because when I'm watching something or I creating a torrent the upload speed drops but drops not for moment and then goes to the normal level, drops for the time when I'm watching the movie or creating the torrent which is normal when there is no cache.

Check MMC's Log for Read/Write Interrupts when qbittorent shows the upload drop, if they exist then whatever the process causing them is at fault.

Do you mean the "Log" from "View" menu in the client?

With what libtorrent version?

Not to be an asshat but the vast majority of windows users will be using the "release" builds as compiling this(most things really) under windows is a fairly tedious procedure until you have it setup..
So this question answers itself alot of the time :X

I'm compiling it with mxe. Under Windows was pain in the ass.

[ciaobaby]

Why while OS cache is enabled happens almost always and while OS cache is disabled - very rarely happens?

You should probably ask Micro$haft for why they haven't fixed their crappy caching system at some point during the last twenty years,  (Ever since NT4 was released in 1996)

The hdd smart is excellent.

Don't rely on S.M.A.R.T. getting it right, it is not that reliable.

[qbi]

Thank you for the answers, boys. Sadly but I'll back to utorrent to see if that problem will appear there.

When OS cache is disabled the buffer size in Statistics is almost the same size like when OS cache is enabled. I'm thinking that when I disable OS cache shouldn't exists buffers and every write/read is directly from/to the hdd without any buffers. How exactly works that buffers and OS cache. In Deluge there is no option "OS cache".

[ciaobaby]

the buffer size in Statistics is almost the same size like when OS cache is enabled

As on would expect, the required "size" does not change, only the mechanism of handling the cached items changes.

How exactly works that buffers and OS cache.

The memory cache is there for holding the 16 kB blocks of data as they arrive from remote peers and are actively being sent to other peers.

. In Deluge there is no option "OS cache".

The option of using either the Operating System or libtorrent to handle the cache was added to qbittorrent so that users where there usual running payloads cause problems  with Windows caching.

Not everybody needs to use libtorrent caching, but when caching causes a problem because of the type of payloads that are running, users have the choice,  the usual cause is high activity, multi-file payloads on Windows operating systems, so if you NEVER run that type of torrent you will probably never come across the problem at all.

[qbi]

So when I disable OS cache actually I enable libtorrent cache?
What I have to change in the libtorrent cache settings so the speed remains in the same level (like when OS cache is enabled) when I'm watching movie or creating torrent?
The libtorrent cache size and cache expiry are ignored by the same settings in qBittorrent, right?
I tried settings for high_performance_seed:
set.cache_size = 32768 * 2;
set.use_read_cache = true;
set.cache_buffer_chunk_size = 128;
set.read_cache_line_size = 32;
set.write_cache_line_size = 32;
set.low_prio_disk = false;
Also I tried cache_line 64, buffer_chunk 254/64, cache_size 65536*2/, 32768 but the speed always drops with OS disabled when I'm playing movie or creating torrents.

[ciaobaby]

when I'm watching movie or creating torrent?

Neither of those are remotely connected to the qbittorrent cache which is a separate entity to the system caching, yes it uses the same hardware (RAM) but the two do not interact especially once the OS cache  is bypassed.

but the speed always drops with OS disabled when I'm playing movie or creating torrents.

"Speeds" dropping when you are 'creating' a torrent is nothing at all to do with caching. It is because libtorrent is single threaded, so when a processor intensive operation  is being done such as hash-summing and committing pieces to permanent storage, creating the meta-data, running a force recheck, moving a payload etc. all other I/O stops until that task is complete, there are two or three threads around here about network transfer speeds dropping to zero during a payload move between file systems or drives.

While watching 'movies' will probably be due to a similar issue, where the media player application has a higher thread priority than qbittorrent/libtorrent, this will possibly be more noticeable if your CPU uses "Hyper-Threading" to simulate 'multiple cores'.

[qbi]

Thanks for the explanation.

but the speed always drops with OS disabled when I'm playing movie or creating torrents.

"Speeds" dropping when you are 'creating' a torrent is nothing at all to do with caching. It is because libtorrent is single threaded, so when a processor intensive operation  is being done such as hash-summing and committing pieces to permanent storage, creating the meta-data, running a force recheck, moving a payload etc. all other I/O stops until that task is complete, there are two or three threads around here about network transfer speeds dropping to zero during a payload move between file systems or drives.

When creating torrent with utorrent with checked "Disable Windows caching of disk reads and writes" there is no speed drop. Is this mean that utorrent protocol is multithreaded?

While watching 'movies' will probably be due to a similar issue, where the media player application has a higher thread priority than qbittorrent/libtorrent, this will possibly be more noticeable if your CPU uses "Hyper-Threading" to simulate 'multiple cores'.

I checked the media player application priority - yes, the default priority is "above normal".

Why there is a min (15s)/max (600s) cache expiry limit in qBittorrent?