Author Topic: Windows tool for migration from Bittorrent/uTorrent/Deluge to qBittorrent  (Read 8987 times)

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
Disappointed in the existing application on the ruby, which could not handle all my torrents, I wrote my own application on golang.
I tested it on my own torrent database (3k+) and test torrents on Bittorrent/uTorrent clients (version 3+).
Features:
  • Processing all torrents
  • Processing torrents with subdirectories and without subdirectories
  • Processing torrents with renamed files
  • Processing torrents with non-standard encodings (for example, cp1251)
  • Processing of torrents in the not ready state *
  • Save date, metrics, status. **
  • Import of tags and labels
  • Multithreading

* This torrents will not be done (0%) and will need force rehash
** The calculation of the completed parts is based only on the priority of the files in torrent
*** Partially downloaded torrents will be visible as 100% completed, but in fact you will need to do a rehash. Without rehash torrents not will be valid. This is due to the fact that conversion of .dat files in which parts of objects are stored is not implemented.

Don't forget before use make backup bittorrent\utorrent, qbittorrent folder. and config %APPDATA%/Roaming/qBittorrent/qBittorrent.ini. Close all this program before.

Usage:
1.) Help (from cmd or powerwhell)
Code: [Select]
C:\Users\user\Downloads> .\bt2qbt_v1.0_amd64.exe -h
Usage of C:\Users\user\Downloads\bt2qbt_v1.0_amd64.exe:
-c, --qconfig (= "C:\\Users\\user\\AppData\\Roaming\\qBittorrent\\qBittorrent.ini")
    qBittorrent config files (for write tags)
-d, --destination (= "C:\\Users\\user\\AppData\\Local\\qBittorrent\\BT_backup\\")
    Destination directory BT_backup (as default)
-s, --source (= "C:\\Users\\user\\AppData\\Roaming\\uTorrent\\")
    Source directory that contains resume.dat and torrents files
--without-labels  (= false)
    Do not export/import labels
--without-tags  (= false)
    Do not export/import tags

2.) If you just run application, it will processing torrents from %APPDATA%\uTorrent\ to %LOCALAPPDATA%\qBittorrent\BT_BACKUP\
Example:
Code: [Select]
It will be performed processing from directory C:\Users\user\AppData\Roaming\uTorrent\ to directory C:\Users\user\AppData\Local\qBittorrent\BT_backup\
Check that the qBittorrent is turned off and the directory C:\Users\user\AppData\Local\qBittorrent\BT_backup\ and config C:\Users\user\AppData\Roaming\qBittorrent\qBittorrent.ini is backed up.


Press Enter to start

Started
1/2 Sucessfully imported 1.torrent
2/2 Sucessfully imported 2.torrent

Press Enter to exit

3.) Run application from cmd or powershell with keys, if you want change source dir or destination dir, or export/import behavior
Example:
Code: [Select]
C:\Users\user\Downloads> .\bt2qbt_v1.0_amd64.exe -s C:\Users\user\AppData\Roaming\BitTorrent\
It will be performed processing from directory C:\Users\user\AppData\Roaming\BitTorrent\ to directory C:\Users\user\AppData\Local\qBittorrent\BT_backup\
Check that the qBittorrent is turned off and the directory C:\Users\user\AppData\Local\qBittorrent\BT_backup\ is backed up.

Press Enter to start
Started
1/3233 Sucessfully imported 1.torrent
2/3233 Sucessfully imported 2.torrent
3/3233 Sucessfully imported 3.torrent
...
3231/3233 Sucessfully imported 3231.torrent
3232/3233 Sucessfully imported 3232.torrent
3233/3233 Sucessfully imported 3233.torrent

Press Enter to exit

Known issuses:
  • Unknown

Here's the code: https://bitbucket.org/rumanzo/bt2qbt/src/master/bt2qbt.go
x86_64 version binary: https://bitbucket.org/rumanzo/bt2qbt/downloads/bt2qbt_v1.0_amd64.exe
i386 version binary: https://bitbucket.org/rumanzo/bt2qbt/downloads/bt2qbt_v1.0_i386.exe

Deluge branch:
code: https://bitbucket.org/rumanzo/deluge2qbt/src/master/deluge2qbt.go
x86_64 version binary: https://bitbucket.org/rumanzo/deluge2qbt/downloads/deluge2qbt_v0.1_amd64.exe
i386 version binary: https://bitbucket.org/rumanzo/deluge2qbt/downloads/deluge2qbt_v0.1_i386.exe
x86_64 linux version binary: https://bitbucket.org/rumanzo/deluge2qbt/downloads/deluge2qbt_linux_v0.1_amd64
i386 linux  version binary: https://bitbucket.org/rumanzo/deluge2qbt/downloads/deluge2qbt_linux_v0.1_i386

If there are any errors or if it help you - write there
« Last Edit: November 11, 2018, 11:43:28 AM by rumanzo »

sledgehammer_999

  • qBittorrent maintainer
  • Administrator
  • Forum addict
  • *****
  • Posts: 2398
  • Karma: +148/-1
    • View Profile
  • Processing torrents with non-standard encodings (for example, cp1251)

Do you mean filename encodings inside the torrent file? If so, what do you do with that?
.torrent files should have utf8 encoding for the filenames. Otherwise, libtorrent replaces invalid utf8 codepoints with an underscore "_". Do you handle this?

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
  • Processing torrents with non-standard encodings (for example, cp1251)

Do you mean filename encodings inside the torrent file? If so, what do you do with that?
.torrent files should have utf8 encoding for the filenames. Otherwise, libtorrent replaces invalid utf8 codepoints with an underscore "_". Do you handle this?
To be honest, I just take the name field name.utf-8 and path.utf-8 in the torrent file if they exists, and in this form I feed to the bencode library. I did not meet any problems, and I did not think that complications could occur. But I can do a check on the validity of the runes (https://golang.org/pkg/unicode/utf8/#Valid).

Despite the fact that the code works nominally, for me there are still things that I do not really understand where to get it  and I would like to rewrite it. But I'm not at all familiar with c/c++ code. If you could tell me a couple of things - that would be fine.

sledgehammer_999

  • qBittorrent maintainer
  • Administrator
  • Forum addict
  • *****
  • Posts: 2398
  • Karma: +148/-1
    • View Profile
Despite the fact that the code works nominally, for me there are still things that I do not really understand where to get it  and I would like to rewrite it. But I'm not at all familiar with c/c++ code. If you could tell me a couple of things - that would be fine.

Ask here, and if I see it I'll try to answer.

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
Despite the fact that the code works nominally, for me there are still things that I do not really understand where to get it  and I would like to rewrite it. But I'm not at all familiar with c/c++ code. If you could tell me a couple of things - that would be fine.

Ask here, and if I see it I'll try to answer.
1.) Will the check runes for utf-8 be correct?
2.) I do not understand how to correctly calculate "blocks per piece" in fastresume. Now I'm using the logic from the previous scripts (on ruby), it works, but I do not like the concept of using fields from BitTorrent resume.dat for this. How to count this field?
3.) I fill the "pieces" field by the number of torrentfile["info"]["lenght"] or sum lenght of files if it multifile torrent / torrentfile["info"]["piece length"] rounded to a larger number. Is it correct?
4.) Now i rehash all my torrents, and some my torrents in which not all files have been selected for downloading, no fully completed. That's ok, bittorrent/utorent create ~BitTorrentPartFile.*.dat, which contain some data, and i not convert it. I am confused that after rehash torrents just stuck. Any idea why this is possible?

sledgehammer_999

  • qBittorrent maintainer
  • Administrator
  • Forum addict
  • *****
  • Posts: 2398
  • Karma: +148/-1
    • View Profile
1.) Will the check runes for utf-8 be correct?

IIRC, normally you don't need to bother with the filenames. You do that if a) the user has renamed the file from inside utorrent b)or has explicitly moved it outside of the folder
In any case, you can't edit the .torrent files because it will change the infohash. The original filenames come from the .torrent. Libtorrent does a sanity check, and replaces invalid utf8 codepoints with and underscore ("_"). And uses this new sanitized string to look for files on disk.

I hope we are talking about the same thing/spot and I am not confusing you.

2.) I do not understand how to correctly calculate "blocks per piece" in fastresume. Now I'm using the logic from the previous scripts (on ruby), it works, but I do not like the concept of using fields from BitTorrent resume.dat for this. How to count this field?

I hope I talking about the same thing.
Unless I am mistaken blocks are always 16KiB. So a piece contains set number of 16KiB blocks. The only exception is the end piece which might contain less data.
I assume that you have to take the piece size from the .torrent file and divide it by 16KiB.

3.) I fill the "pieces" field by the number of torrentfile["info"]["lenght"] or sum lenght of files if it multifile torrent / torrentfile["info"]["piece length"] rounded to a larger number. Is it correct?

I think not. Maybe the calculation of the number of pieces is correct. The data you fill the piece array might not be correct.
Some more info:
Here's an old explanation by me on what I had found on resume.dat when I experimented with it. Link to specific comment. You might want to read the thread from the start to get any other useful info.
And from libtorrent docs here's an explanation on what various fields in the fastresume do: https://libtorrent.org/manual-ref.html#fast-resume

4.) Now i rehash all my torrents, and some my torrents in which not all files have been selected for downloading, no fully completed. That's ok, bittorrent/utorent create ~BitTorrentPartFile.*.dat, which contain some data, and i not convert it. I am confused that after rehash torrents just stuck. Any idea why this is possible?

Maybe it is in weird state, or you just don't have any peers anymore?

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
Quote
IIRC, normally you don't need to bother with the filenames. You do that if a) the user has renamed the file from inside utorrent b)or has explicitly moved it outside of the folder
In any case, you can't edit the .torrent files because it will change the infohash. The original filenames come from the .torrent. Libtorrent does a sanity check, and replaces invalid utf8 codepoints with and underscore ("_"). And uses this new sanitized string to look for files on disk.

I hope we are talking about the same thing/spot and I am not confusing you.
It looks like I do not really need to worry about this.

Quote
I hope I talking about the same thing.
Unless I am mistaken blocks are always 16KiB. So a piece contains set number of 16KiB blocks. The only exception is the end piece which might contain less data.
I assume that you have to take the piece size from the .torrent file and divide it by 16KiB.
It is! So simple, thanks.

Quote
I think not. Maybe the calculation of the number of pieces is correct. The data you fill the piece array might not be correct.
Some more info:
Here's an old explanation by me on what I had found on resume.dat when I experimented with it. Link to specific comment. You might want to read the thread from the start to get any other useful info.

And from libtorrent docs here's an explanation on what various fields in the fastresume do: https://libtorrent.org/manual-ref.html#fast-resume

Thanks for links. But I'm not sure that we're talking about the same thing. I mean the length of the "pieces" field. I fill it with 0 or 1 (0x00 or 0x01), it's correct. If calculation of the number of pieces is correct, it's all right.

Quote
Maybe it is in weird state, or you just don't have any peers anymore?
This does not apply to all torrents, and in the log, I see a io error ... error: permission denied. Apparently this is due not to the torrents themselves, but to something else. I'll try to figure out the other day.


P.S. I try to modify top message and got error:
Forbidden
You don't have permission to access /index.php on this server.
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
« Last Edit: May 06, 2018, 04:22:14 PM by rumanzo »

sledgehammer_999

  • qBittorrent maintainer
  • Administrator
  • Forum addict
  • *****
  • Posts: 2398
  • Karma: +148/-1
    • View Profile
Thanks for links. But I'm not sure that we're talking about the same thing. I mean the length of the "pieces" field. I fill it with 0 or 1 (0x00 or 0x01), it's correct. If calculation of the number of pieces is correct, it's all right.

IMO, the length should be taken from the .torrent file. It has a "pieces" field too.
About the "pieces" field in the fastresume: That's a byte array (or byte string). Each element is one byte(character) and represents one piece. Each byte has 8bits. Setting the 1st bit means we have that piece. Setting the 2nd means that we verified that piece aka it passed the hash check. If that piece is unverified, it usually gets hash-checked during seeding when it is requested for upload.
So, if I am not mistaken the correct values are:
1. 0x0 -> piece not downloaded
2. 0x80 -> piece downloaded but not verified
3. 0xc0 -> piece downloaded and verified
4. 0x40 -> I don't think this is valid. It means piece not downloaded but verified.

(those values were obtained from online binary to hex converters)

This does not apply to all torrents, and in the log, I see a io error ... error: permission denied. Apparently this is due not to the torrents themselves, but to something else. I'll try to figure out the other day.

Then that probably means that something's wrong with the conversion.

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
IMO, the length should be taken from the .torrent file. It has a "pieces" field too.
It has, but I can not find the connection between that number and what should be.
For example:
In torrent file - piece lenght = 8388608, len pieces - 105960
In fastresume file - len pieces - 5298.
It's seems like len(fr[pieces]) = len(tr[pieces])/20. But i don't sure and don't understand why.
About the "pieces" field in the fastresume: That's a byte array (or byte string). Each element is one byte(character) and represents one piece. Each byte has 8bits. Setting the 1st bit means we have that piece. Setting the 2nd means that we verified that piece aka it passed the hash check. If that piece is unverified, it usually gets hash-checked during seeding when it is requested for upload.
Correct.

Then that probably means that something's wrong with the conversion.
It's sad. Will debug this.

sledgehammer_999

  • qBittorrent maintainer
  • Administrator
  • Forum addict
  • *****
  • Posts: 2398
  • Karma: +148/-1
    • View Profile
It has, but I can not find the connection between that number and what should be.
For example:
In torrent file - piece lenght = 8388608, len pieces - 105960
In fastresume file - len pieces - 5298.
It's seems like len(fr[pieces]) = len(tr[pieces])/20. But i don't sure and don't understand why.

It seems I was slightly wrong.
The bittorrent BEP says this about the "pieces" field:
Quote
pieces maps to a string whose length is a multiple of 20. It is to be subdivided into strings of length 20, each of which is the SHA1 hash of the piece at the corresponding index.

And that's why you need to divide by 20 in your example.

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
It has, but I can not find the connection between that number and what should be.
For example:
In torrent file - piece lenght = 8388608, len pieces - 105960
In fastresume file - len pieces - 5298.
It's seems like len(fr[pieces]) = len(tr[pieces])/20. But i don't sure and don't understand why.

It seems I was slightly wrong.
The bittorrent BEP says this about the "pieces" field:
Quote
pieces maps to a string whose length is a multiple of 20. It is to be subdivided into strings of length 20, each of which is the SHA1 hash of the piece at the corresponding index.

And that's why you need to divide by 20 in your example.

Thanks!

And I found problem with my torrents - why then on some files there was an attribute Read-only.
And I still can't modify top message, who can help with that?
Forbidden
You don't have permission to access /index.php on this server.
Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

sledgehammer_999

  • qBittorrent maintainer
  • Administrator
  • Forum addict
  • *****
  • Posts: 2398
  • Karma: +148/-1
    • View Profile
Do you click the "Modify" button in your first post and you have those errors?

rumanzo

  • Member
  • **
  • Posts: 39
  • Karma: +4/-0
    • View Profile
Do you click the "Modify" button in your first post and you have those errors?
Yes, if i try modify throw top button "Modify" a get this error. If i try use down button, nothing happens.

Peter

  • Administrator
  • Forum addict
  • *****
  • Posts: 1552
  • Karma: +37/-2
    • View Profile
Do you click the "Modify" button in your first post and you have those errors?
Yes, if i try modify throw top button "Modify" a get this error. If i try use down button, nothing happens.

Can you please PM me the error you get when you try Modify?
Thank you!
- qBittorrent team - server and forum administrator.
- Hungarian translation reviewer/moderator (+ translator).

Join the official qBittorrent Discord!
https://discord.gg/ma66Vv4

Gun

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Hi, getting this error with 0.6 x64

Code: [Select]
panic: interface conversion: interface {} is []interface {}, not string

goroutine 263 [running]:
main.(*NewTorrentStructure).gettrackers(0xc043cae2c0, 0xc042b47f00, 0x2, 0x4)
        C:/Users/rumanzo/eclipse-workspace/testinggo/src/bt2qbt/bt2qbt.go:212 +0x1e9
main.logic(0xc042b31120, 0x1e, 0xc042a590e0, 0xc04204c2f0, 0xc04200e180, 0xc04200e181, 0xc04204c300, 0xc042c6a2a0, 0x1af, 0xc04207dc90, ...)
        C:/Users/rumanzo/eclipse-workspace/testinggo/src/bt2qbt/bt2qbt.go:407 +0xa79
created by main.main
        C:/Users/rumanzo/eclipse-workspace/testinggo/src/bt2qbt/bt2qbt.go:510 +0xcd4

In 0.4 it looks like this

Code: [Select]
panic: runtime error: index out of range

goroutine 275 [running]:
main.logic(0xc042d11630, 0x42, 0xc042cf3140, 0xc04204c300, 0xc04200e180, 0xc04200e181, 0xc04204c310, 0xc042c6fe00, 0x1af, 0xc04207dc90, ...)
        /home/kostinan/1T/testinggo/src/bt2qbt/bt2qbt.go:261 +0x5d40
created by main.main
        /home/kostinan/1T/testinggo/src/bt2qbt/bt2qbt.go:414 +0xcd4