network freezes, nox 4.3.3 on ubuntu 20.04

Linux specific questions, problems.
Post Reply
vesoljc

network freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

Hi, after upgrading from 4.1.3 to 4.3.3 (I think ubuntu server was bumped also), my home server is having random but regular freezes. The freeze lasts for 5-30seconds, and randomly repeats (usually 5-10 minutes). During that time, connection to the server is down, ping fails, samba share breaks, webui for qbit is down. If I'm logged on directly (keyboard & monitor) to the server, the CLI is responsive, but network seems to be down (pinging my dhcp router fails). Also, during freeze, webmin stats show this (notice the bump at 12:33):
tw2U5.png
tw2U5.png (91.87 KiB) Viewed 3765 times
I did some diagnostics on the shared disks (samba on mdraid 3x3TB) but all seems fine.
I also tried replacing a NIC with a spare, changed the network cable, but it's the same.


Funny things, if I disable the qbittorrent-nox service, those freezes are gone. I've checked the log, but didn't notice anything weird.
And ideas on what might be the issue? What else to check?

UPDATE:
Figured out that it's not system wise freeze, just a network stack issue.
Last edited by vesoljc on Fri Mar 19, 2021 10:58 am, edited 1 time in total.
unseen19
Veteran
Veteran
Posts: 188
Joined: Mon Mar 15, 2021 1:36 pm

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by unseen19 »

Hi,

dmesg/journalctl point to any error?
No oom kill ?
vesoljc

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

I think I'm getting a crash, network interface related...
I've update the driver with the latest from realtek, but it's the same. I was thinking that maybe it's not directly qb related, maybe it's just network traffic stress :)

So, I disabled the nox service and started moving 25GB of data up and down to saturate the bandwidth (ssd to ssd), but to my suprise, even with fully saturated network, it kept working fine. ???

Once I start the service again, problems reappear...
Attachments
logs.zip
(49.11 KiB) Downloaded 173 times
unseen19
Veteran
Veteran
Posts: 188
Joined: Mon Mar 15, 2021 1:36 pm

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by unseen19 »

It still could be the stress but the transfer will not show it: moving files is not the same behaviour as having hundred connected to torrent pieces
Connection number vs bandwidth to say it simply.

May you try to reduce the total connection number to let's say 300 total in QBT-nox?
A netstat (ss) may give you the picture too, bmon may help check the packet/s (and can record)
vesoljc

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

I tried reducing number of connections, still same end result.

My linux skills are low, can you point me at what should be looking at?
netstat.zip
(3.73 KiB) Downloaded 152 times
unseen19
Veteran
Veteran
Posts: 188
Joined: Mon Mar 15, 2021 1:36 pm

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by unseen19 »

Well connections number is not the problem following the log.

Let's take it from scratch:
- what's the server hardware?
- how many torrents?
-Only occurring since the update, was the libtorrent version updated too? My test setup is on RPI4 QBT 4.3.3 and libtorrent 1.2.12. Version may help Googling the issue for similar pattern
-Are you able to monitor your (with the graph in your 1st post- the memory and swap usage? If it occurs the swap may slow down the whole server. Otherwise htop, free -h, and similar may help.
-in the advanced setting try to reduce to something small the disk cache. This will help with the previous point too
- is it related to download action or simply seeding torrent make it happen too?
- are you able to test with a direct attached disk? If I'm not wrong your are using a samba share as target. this will dodge the possible memory read/write staging and the associated massive write it may come with.

To be clear it's just some tests I'm thinking about first. The memory misbehavior may explain things but it's more guessing at this level.
The direct attached disk, even simple usb, may be a good test to way to start
vesoljc

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

msi b250 board, celeron 3930, 16gb ram, 250gb nvme for os, 3x3TB (wd red) in mirror mdraid, realtek 2.5gb NIC. i've been using this setup for about 2years now, didn't have any real issues once I set it up. I upgraded the ubuntu server a couple of times (16 -> 18 -> 20).

number of torrents is really low, I'm currently seeding 3, downloading 1.

yep, this started last week, after I updated a few things (nox to 4.3.3, i think i was on 4.1 prior). I installed the nox via apt, so I presume it's using latest libtorrent?

The memory usage during these freezes doesn't seem to be affected, swap is constantly low, real memory usage is showing 14%. I ran htop, IO rates seem normal, cpu usage also, only memory seems a bit high? 2GB? Is this ok, not sure what are "normal" numbers?
htop01.png
htop01.png (48.18 KiB) Viewed 3743 times
I just tried downloading to nvme ssd and the freeze again happened.
unseen19
Veteran
Veteran
Posts: 188
Joined: Mon Mar 15, 2021 1:36 pm

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by unseen19 »

Well regarding htop and the hardware you use I can hardly support hypothesis I made previously. Ubuntu has not yet the 4.3.3 version so far I know so I presume you added the PPA yourself? Again here the libtorrent is the backbone Qbittorrent relies on, what tells the interface about its version?
A "dpkg -l|grep -i libtorrent" will work too.

I've test machine in 20.04lts and 20.10 (nox and ubuntu-desktop so) and cannot confirm the behavior. Test machines only so with no more than 10 torrents each however but still with no more people yelling about performance the issue should be local.
vesoljc

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

yep, i've added repo by hand

dump:
ii libtorrent-dev:amd64 0.13.8-2build1 amd64 C++ BitTorrent library by Rakshasa (development files)
ii libtorrent-rasterbar10 1.2.12+git20210118.3efdb8a941-1ppa1~20.04 amd64 C++ bittorrent library by Rasterbar Software
ii libtorrent21:amd64 0.13.8-2build1 amd64 C++ BitTorrent library by Rakshasa
ii qbittorrent-nox 1:4.3.3.99~202101191832-7248-da0b276d5~ubuntu20.04.1 amd64 bittorrent client based on libtorrent-rasterbar (without X support)

so, I guess I'm still on 1.2...
build it by hand?
unseen19
Veteran
Veteran
Posts: 188
Joined: Mon Mar 15, 2021 1:36 pm

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by unseen19 »

That fine, we see qbittorrent-nox 4.3.3 and livtorrent 1.2.12.
I use the same version currently

Let's put the hardware away, nothing obvious with what I see. The behaviour changed with an apt upgrade, OS and QBT update thanks the PPA
Save your setting first and:

-First update the libtorrent version. I just saw the PPA has an updates version from 08 March
libtorrent-rasterbar 1.2.13+git20210308.ed960c2562-1ppa1
Stop the service first, update, restart or reboot and tell us. This is an easy change

-if not working, remove the PPA from apt source, uninstall QBT/libtorrent, and reinstall from ubuntu repo. Ubunti 20.04 still has QBT 4.1.7 in the official repo.

I hope it will help to sort things out. At least I can tell you the qbt-nox on my test 20.04lts box doesn't give the same freeze with the exact same version you have now. My 20.10 ARM neither.
vesoljc

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

libtorrent-rasterbar10 is already the newest version (1.2.12+git20210118.3efdb8a941-1ppa1~20.04).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.


How exactly do I upgrade it?
vesoljc

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

Am I correct to assume that:
1.2.13+git20210308.ed960c2562-1ppa1~20.10

requires 20.10 ubuntu?
unseen19
Veteran
Veteran
Posts: 188
Joined: Mon Mar 15, 2021 1:36 pm

Re: system freezes, nox 4.3.3 on ubuntu 20.04

Post by unseen19 »

Starting my post by a bunch on config check but...trying myself and I cannot retrieve the 1.2.13 either.
Well listed on the PPA but I just noticed the build failed for all platforms!
https://launchpad.net/~qbittorrent-team ... ate=failed
vesoljc

Re: network freezes, nox 4.3.3 on ubuntu 20.04

Post by vesoljc »

Well, to be honest, now it seems to be fixed. Problem is, I'm not sure what fixed it. There were some updates on ubuntu and I've changed the NIC, again, this time to a 10g. I've had some issues with it, so I reconfigured the network conf and now it magically works... At least with the current limited testing. But the nox was up all night/day, ping to the server is stable, samba share works fine.
Will post updates...
gunyalizop

Re: network freezes, nox 4.3.3 on ubuntu 20.04

Post by gunyalizop »

It still could be the stress but the transfer will not show it: moving files is not the same behaviour as having hundred connected to torrent pieces
Connection number vs bandwidth to say it simply.
Post Reply