Author Topic: Torrent read/write nvme benchmark.  (Read 1362 times)

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Torrent read/write nvme benchmark.
« on: July 05, 2019, 04:20:17 PM »
This has been talked about before, but most of the posts are scattered around the forum and can be difficult to find by search. And google isn't of much help here or i am using the wrong keywords.

I have been running raid0 ssd's as temp for quite some time now, it has performed very well. But regardless, it is not quite fast enough to handle simultaneous +600mbit with ~100 connections.
Nvme drives have become a lot cheaper and the motherboard support easily available. Switching the raid0 for a nvme drive would be the next logical step.
What i am unsure about is which type of benchmark you should look for when you know the drive would be used for torrents. Crystaldiskmark benches mostly 4kb and doesn't do simultaneous benchmark. IOmeter is another option, but don't have a lot of experience with that, and there's many other tools. But which would provide the most relatable benchmark for torrent use?

Switeck

  • Forum addict
  • ****
  • Posts: 1518
  • Karma: +96/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #1 on: July 06, 2019, 06:02:40 PM »
I've been downloading torrents to a multi-GB sized ramdrive for years. It's set up to move the finished torrents to a HDD after it's done.
But even ramdrives have their speed limitations.
Ramdrive software is often poorly written, because they're not made by billion-dollar corporations that can have a team of elite programmers that work years on the task.

At the top end, a PCI Express based SSD "enterprise class" (or better) will beat a ramdrive for most uses simply because the ramdrive puts too much demands on the CPU.
Most ramdrive software is single-threaded and cannot use multi-core CPUs very well -- at their max speed, they max out one CPU core and that's it.
The really fast SSDs that utilize the PCI Express bus can do direct memory access and be much easier on the CPU.

More about Nvme SSDs:
https://en.wikipedia.org/wiki/NVM_Express

In short, the best Serial ATA 3 (SATA-3) based SSDs probably max out around 400-500 MB/sec speeds for very sequential data reads/writes.
Ramdrives can max out potentially over 10000 MB/sec (10 GB/sec!), but CPU usage goes way up when they're doing that.
Top-end PCI Express based SSDs might reach over 5000 MB/sec, but put much less load on the CPU.

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #2 on: July 06, 2019, 09:44:41 PM »
I've been downloading torrents to a multi-GB sized ramdrive for years. It's set up to move the finished torrents to a HDD after it's done.
But even ramdrives have their speed limitations.
Ramdrive software is often poorly written, because they're not made by billion-dollar corporations that can have a team of elite programmers that work years on the task.

Ramdrive isn't really an option, i have checked and +128gb memory is super expensive, even if you're looking for used parts. :)

At the top end, a PCI Express based SSD "enterprise class" (or better) will beat a ramdrive for most uses simply because the ramdrive puts too much demands on the CPU.
Most ramdrive software is single-threaded and cannot use multi-core CPUs very well -- at their max speed, they max out one CPU core and that's it.
The really fast SSDs that utilize the PCI Express bus can do direct memory access and be much easier on the CPU.

This is why i am looking at nvme, it's fast and it's getting cheaper. But is it fast in the areas where i want it to be fast, is it faster than two ssd's in raid0 for torrent usage. My raid0 can do ~1gb/s as shown below, i have no use for that as it simply cannot keep up with the iops from a torrent at gbit speeds, and that's only 1/10 of the speed.



I did not know about the difference in cpu utilization, that's good to know and could also be a reason to switch to nvme if that is any issue one has.

In short, the best Serial ATA 3 (SATA-3) based SSDs probably max out around 400-500 MB/sec speeds for very sequential data reads/writes.
Ramdrives can max out potentially over 10000 MB/sec (10 GB/sec!), but CPU usage goes way up when they're doing that.
Top-end PCI Express based SSDs might reach over 5000 MB/sec, but put much less load on the CPU.

I don't think the interface speed is very important, sata3 is plenty fast. What matters is the device attached to the interface. An HDD would be able to fully utilize a 5/5mbit connection, but you start to increase the bandwidth and the iops would quickly overload the drive. The attached device is the limiting factor, not the interface.
What's valuable in a drive for torrent use, is it's iops performance at the data sizes torrents use, which i don't remember, and therefore unsure which type of benchmark you should look for, that best translates into torrent performance.
« Last Edit: July 07, 2019, 01:30:33 AM by fusk »

Switeck

  • Forum addict
  • ****
  • Posts: 1518
  • Karma: +96/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #3 on: July 07, 2019, 02:42:44 PM »
I can only download "smaller" torrents to my ramdrive. The rest get downloaded straight to a HDD.
But my connection is only ~1/10th as fast as yours on download speed. Upload is considerably slower than that.

I may have overestimated top speeds for mid-to-high end NVME SSDs at this time, but ramdrives can crank out >1 GB/sec even on very fragmented data...the CPU has to be fast to manage that though!

qBitTorrent post ~v4.0.something should be able to do coalesce writes and reads on HDDs.
So long as there's a big enough sequential run, the data pattern will probably resemble something between 4 KB and 1 MB group reads/writes.

You may have done so much downloading on your Raid 0 SSD that Trim is slowing it down.

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #4 on: July 07, 2019, 05:23:38 PM »
I can only download "smaller" torrents to my ramdrive. The rest get downloaded straight to a HDD.
But my connection is only ~1/10th as fast as yours on download speed. Upload is considerably slower than that.

I may have overestimated top speeds for mid-to-high end NVME SSDs at this time, but ramdrives can crank out >1 GB/sec even on very fragmented data...the CPU has to be fast to manage that though!

qBitTorrent post ~v4.0.something should be able to do coalesce writes and reads on HDDs.
So long as there's a big enough sequential run, the data pattern will probably resemble something between 4 KB and 1 MB group reads/writes.

You may have done so much downloading on your Raid 0 SSD that Trim is slowing it down.

Ramdrive would be cool, and simultaneous down/up at +900mbit is probably fine. But +128gb is just waaay too expensive, so i am not even considering going that route.

4kb hurts performance quite a lot. I've seen many people mention that torrent traffic relies heavily on 16kb random read/write, i don't know if that's mostly swarm -> memory, and memory -> storage mostly being entire pieces. I've noticed that most torrents these days are created with a piece size of 1mb or higher. But how that affects transfer from memory -> storage i do not know.
I saw a post on libtorrent github where someone talked about downloading entire pieces to memory before transferring to storage if possible, as storage would have an easier time keeping up, and would allow for higher transfer speeds. But that was something for a future update.

I have not noticed the raid0 getting slower, and while it is fast, it's not fast enough. IOmeter can do 16kb benchmarks, but not simultaneous so those numbers wouldn't reflect real world performance. The only benchmark i've found so far that does simultaneous test is passmark's performancetest which has a disk test that you can manually set up, but only the file size and cluster size can be set, which doesn't help much if you want to bench 16kb random read/write.
« Last Edit: July 07, 2019, 05:28:12 PM by fusk »

Switeck

  • Forum addict
  • ****
  • Posts: 1518
  • Karma: +96/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #5 on: July 07, 2019, 05:57:45 PM »
There is no need to have +128 GB of ram for torrenting from/to a ramdrive.
My ramdrive is typically set to just shy of 7 GB size -- and most of the torrents I get easily fit inside that.
I'm unlikely to be downloading more than 1 torrent larger than that at a time.

16KB is the chunk size inside pieces that BitTorrent natively deals with, compounded by the problems created by multi-file torrents where pieces get misaligned relative to file bytes. (With the start of the 2nd file often being mid-piece.)

The future update is probably libtorrent-based cache improvements, which qBitTorrent will likely take advantage of when it becomes available. Aligning disk writes with file block locations may already be done, which should help some with SSD write amplification issues.

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #6 on: July 07, 2019, 07:06:58 PM »
Over 60% of my torrents are over 7gb, and if the ramdrive is skipped it would download to main storage which is mechanical and that's gonna be a bad time. A 60gb ramdrive would fit ~80% of my torrents, 128gb would fit 95%. I often download more than 1 at a time. Optane is another idea, but won't work on a ryzen platform.
It would be cheaper to get a really fast 256gb nvme drive instead, while it won't perform the same, it can perform better than the raid0 ssd's i use as temp right now. If i had the space i would even consider raid0 nvme. But i am having a hard time finding out what the performance difference means in the real world, because 4kb benches shows some nvme to only be ~15/20mb/s faster than mine for read and almost double for write, but those are not simultaneous benchmarks. Maybe it's enough to make a huge difference.

At Q1T2(T2 because qbit is set to use 2 threads) my drive does.
« Last Edit: July 07, 2019, 08:23:55 PM by fusk »

Switeck

  • Forum addict
  • ****
  • Posts: 1518
  • Karma: +96/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #7 on: July 07, 2019, 10:08:07 PM »
I can get away with downloading directly to HDD because the torrents I go after are smaller and more specifically typically SLOW due to being public torrents with few seeds+peers.
This isn't to say I download only 1 torrent at a time, just that I seldom have more than 1 downloading that's bigger than will fit in the ramdrive.
Almost everything finishes faster than I can seed back to 1:1, so I don't have a download list piling up.

A really fast 256gb nvme drive does indeed sound like the best choice for you, but "really fast" may be rather expensive.

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #8 on: July 08, 2019, 01:00:34 AM »
I can get away with downloading directly to HDD because the torrents I go after are smaller and more specifically typically SLOW due to being public torrents with few seeds+peers.
This isn't to say I download only 1 torrent at a time, just that I seldom have more than 1 downloading that's bigger than will fit in the ramdrive.
Almost everything finishes faster than I can seed back to 1:1, so I don't have a download list piling up.

A really fast 256gb nvme drive does indeed sound like the best choice for you, but "really fast" may be rather expensive.

Public trackers would probably make a big difference, a typical torrent for me overloads my raid0 ssd's and i need to limit download speeds to ~49mb/s to not kill upload. Not having see any reviews, but 970 EVO Plus according to https://ssd.userbenchmark.com/ is the fastest 256gb at 4k mixed read/write and it's ~84€, 500gb is 117€ and slightly faster.
And i could probably sell the 2 evo 850's for 30€ each.
« Last Edit: July 08, 2019, 01:46:16 AM by fusk »

buratino

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #9 on: July 08, 2019, 09:08:45 AM »
I don't know but unless you own a real 10Gbps internet connection, a normal single SSD should suffice.
My experience is this. I have a true 1Gbps internet connection and save torrents to a sata SSD. Absolutely no issues, at least for 20-30 torrents with 2000 max connections set. Sure, most torrents I download are single big files (10-30GB). Almost the same story when I save to HDD (a big spinner at 210MB/s max linear speeds).
The limiting factor in my experiments with several torrent clients was the CPU and single-threadiness of most clients. Things like tixati apparently utilize one core and hit a limit at certain point.
After many sotwares I settled down with qBittorrent which is doing its job beatifully and I can see steady 110+MB/s over an entire session of downloading 150-200GB.
As a result I never thought of building raids or use nvme for torrents.

Switeck

  • Forum addict
  • ****
  • Posts: 1518
  • Karma: +96/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #10 on: July 08, 2019, 10:36:29 AM »
fusk, since you need a BIG jump in speed to stay somewhat future-proof, I'd strongly recommend the 500gb for 117€.

Another important thing to mention on why I don't experience bad torrent problems is I mostly use uTorrent (v3.5.5 now) -- it's much better at handling file reads/writes than libtorrent does in qBitTorrent.
There's work being done on the issue in libtorrent, but it's mostly experimental/alpha stage at the moment.

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #11 on: July 08, 2019, 01:57:47 PM »
Another important thing to mention on why I don't experience bad torrent problems is I mostly use uTorrent (v3.5.5 now) -- it's much better at handling file reads/writes than libtorrent does in qBitTorrent.
There's work being done on the issue in libtorrent, but it's mostly experimental/alpha stage at the moment.

I see, i knew there were some issues with libtorrent, afaik it was noticeable issue, but not a major one. When you say "much better" what are we talking here?

I don't know but unless you own a real 10Gbps internet connection, a normal single SSD should suffice.
My experience is this. I have a true 1Gbps internet connection and save torrents to a sata SSD. Absolutely no issues, at least for 20-30 torrents with 2000 max connections set. Sure, most torrents I download are single big files (10-30GB). Almost the same story when I save to HDD (a big spinner at 210MB/s max linear speeds).
The limiting factor in my experiments with several torrent clients was the CPU and single-threadiness of most clients. Things like tixati apparently utilize one core and hit a limit at certain point.
After many sotwares I settled down with qBittorrent which is doing its job beatifully and I can see steady 110+MB/s over an entire session of downloading 150-200GB.
As a result I never thought of building raids or use nvme for torrents.

While that might be correct for you, it isn't for me. There's a reason why even "slower" 1gbit seedboxes also use ramdrives, ssd raids etc. Downloading a single file torrent with 15 connections and everything is probably fine as you said.
But try and jump on a new torrent less than a min after it's added to a private tracker, try 60 down/up slots and your drive will be thrashed. The reason being the amount of small sized iops at gbit speeds, it's the thing storage has the hardest time dealing with, and that's what the torrent protocol mostly uses.
Reference: https://qbforums.shiki.hu/index.php/topic,2627.0.html

Downloading is not what this is about, downloading and uploading simultaneous at gbit speeds is what this is about. If you're a casual user, this isn't meant for you.
« Last Edit: July 08, 2019, 03:00:51 PM by fusk »

buratino

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #12 on: July 08, 2019, 06:39:34 PM »
Ok, then I guess this affects initial torrent uploaders, the first few minutes of seeding.
I was talking about 20-30 torrents with 2000 connections limit (average of 50-100 per torrent). Most of them are new and have reasonable upload. So, 1000 files per torrent with 20MB/s or 2 files per torrent with 20MB/s, torrents transfer pieces, so it's comparable. All pieces fight for the available bandwidth limit.
If some seedboxes use ramdrives, it's for other reasons.
But yes, other than that I'm a casual user, as you like.

fusk

  • Veteran
  • ***
  • Posts: 248
  • Karma: +4/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #13 on: July 08, 2019, 10:58:21 PM »
If it was seeding you'd only be interested in read speeds. At present i have 266 torrents in the list.
« Last Edit: July 08, 2019, 11:06:34 PM by fusk »

Switeck

  • Forum addict
  • ****
  • Posts: 1518
  • Karma: +96/-0
    • View Profile
Re: Torrent read/write nvme benchmark.
« Reply #14 on: July 09, 2019, 12:59:45 AM »
Another important thing to mention on why I don't experience bad torrent problems is I mostly use uTorrent (v3.5.5 now) -- it's much better at handling file reads/writes than libtorrent does in qBitTorrent.
I see, i knew there were some issues with libtorrent, afaik it was noticeable issue, but not a major one. When you say "much better" what are we talking here?
uTorrent has been saving whole pieces in its cache and writing them to disk in 1 go for over a decade. qBitTorrent still doesn't seem to do that -- it'll just write maybe 512 KB at once.