Making FreeBSD magnet links

Updated: December 7, 2022 at 12:20 PM

Originally Posted: July 3, 2018 at 4:49 PM

Note: This was originally posted here, but things have been improved since then and so it has been updated.

For the last few years, I’ve been producing torrents and publishing magnet links, but there is some special work that I do to make these. The first few releases, I inserted a bogus tracker into the torrent, because despite there being plenty of tools out there for producing trackerless (DHT) torrents, they were all GUI and I never found any that were command line based. The other was there was/is no tool for extracting the info hash and building the magnet link. There may be tools now, but I couldn’t find any when I started 3 years ago.

The following steps are based upon the recent release of FreeBSD 11.2‑R, adjust as necessary.

  1. Fetch the FreeBSD release and release announcement into a directory (I create a per release directory). remove the CHECKSUM.SHA256, CHECKSUM.SHA512 and index.html* files.
    
       $ REL=12.4-RELEASE
       $ RELABV=12.4R
       $ curl https://www.funkthat.com/~jmg/FreeBSD-snap/snapshot.idx.xz | xzcat | awk ' $2 == "'"$REL"'" { print $9 }' | xargs -L 1 -P 3  wget --no-verbose -c --limit-rate=7000k
       $ wget https://www.freebsd.org/releases/$RELABV/announce.asc
    
       
    This step depends upon snapaid running. If it is no longer running for some reason, addinfo.sh could be run manually against the release email (not the downloaded announce.asc file as the message id is needed) to generate snapshot.idx.xz locally, and use that instead.
  2. Verify the GPG key that signed the above files. This is usually Glen Barber’s key, but not always. I have met and verified his fingerprint in person, If you have verified someone’s key who has signed Glen’s key, that is another good way.
  3. Verify the release announcement:
    
       $ gpg --verify announce.asc
       Warning: using insecure memory!
       gpg: Signature made Mon Dec  5 17:34:59 2022 PST using RSA key ID 478FE293
       gpg: Good signature from "Glen Barber <gjb @ FreeBSD.org>" [unknown]
       gpg:                 aka "Glen Barber <gjb @ keybase.io>" [unknown]
       gpg:                 aka "Glen Barber <gjb @ glenbarber.us>" [unknown]
       gpg:                 aka "Glen Barber <glen.j.barber @ gmail.com>" [unknown]
       gpg: WARNING: This key is not certified with a trusted signature!
       gpg:          There is no indication that the signature belongs to the owner.
       Primary key fingerprint: 78B3 42BA 26C7 B2AC 681E  A7BE 524F 0C37 A0B9 46A3
            Subkey fingerprint: 8D12 403C 2E6C AB08 6CF6  4DA3 0314 58A5 478F E293
       
  4. In the past I have used BitTornado for other things, so I ended up using it as the basis to make the tool for creating trackerless torrent files. The modifications were simple. It appears that the original BitTornado CVS tree is off-line (anyways, it was served insecurely), but it looks like effigies/BitTornado is similar enough that it could be modified and used. I copied btmakemetafile.py to btmaketrackerless.py and applied the following patch:
    
       $ diff -u btmakemetafile.py btmaketrackerless.py 
       --- btmakemetafile.py   2004-05-24 12:54:52.000000000 -0700
       +++ btmaketrackerless.py        2016-10-10 17:13:32.742081000 -0700
       @@ -23,9 +23,9 @@
        def prog(amount):
            print '%.1f%% complete\r' % (amount * 100),
    
       -if len(argv) < 3:
       +if len(argv) < 2:
            a,b = split(argv[0])
       -    print 'Usage: ' + b + ' <trackerurl> <file> [file...] [params...]'
       +    print 'Usage: ' + b + ' <file> [file...] [params...]'
            print
            print formatDefinitions(defaults, 80)
            print_announcelist_details()
       @@ -33,9 +33,9 @@
            exit(2)
    
        try:
       -    config, args = parseargs(argv[1:], defaults, 2, None)
       -    for file in args[1:]:
       -        make_meta_file(file, args[0], config, progress = prog)
       +    config, args = parseargs(argv[1:], defaults, 1, None)
       +    for file in args[0:]:
       +        make_meta_file(file, None, config, progress = prog)
        except ValueError, e:
            print 'error: ' + str(e)
            print 'run with no args for parameter explanations'
    
       
    If you notice, the only thing that is done is to drop the first argument, and instead of passing it into make_meta_file, a None is passed instead. This will simply not add trackers to the torrent file.
  5. I then run the following script to verify the downloaded files, and generate the torrent files:
    
       $ cat cmp.sh
       #!/bin/sh -
       # REL=12.4-RELEASE
       # RELABV=12.4R
       # xzcat ~jmg/public_html/FreeBSD-snap/snapshot.idx.xz | awk ' $2 == "'"$REL"'" { print $9 }' | xargs -L 1 -P 3  wget --no-verbose -c --limit-rate=7000k
       # XXX - following command 404, manually saved from relase announcement
       # wget https://www.freebsd.org/releases/$RELABV/announce.asc
       # wget -c https://download.freebsd.org/ftp/releases/CI-IMAGES/$REL/amd64/Latest/FreeBSD-$REL-amd64-BASIC-CI.raw.xz
       # wget -c https://download.freebsd.org/ftp/releases/VM-IMAGES/$REL/riscv64/Latest/FreeBSD-$REL-riscv-riscv64.{qcow2,raw,vhd,vmdk}.xz
    
       grep -h '^   SHA512' announce.asc | sed -e 's/SHA512 (\(.*\)) = \(.*\)/\2 \1/' | sort -k 2 > sha512.from.asc
    
       while read hash fname; do
               if [ -e "$fname" ]; then
                       if [ -e "$fname".torrent ]; then
                               echo skipping "$fname"...
                               continue
                       fi
    
                       sigfile=$(grep -l -- "$fname" *.asc | head -n 1)
                       echo checking "$fname", sig in: "$sigfile"
                       #res=`sha512 -q "$fname"`
                       #res=`shasum -a 512 "$fname" | awk '{ print $1 }'`
                       res=$(openssl sha512  < "$fname" | awk '{ print $2 }')
                       echo "File is: $res"
                       if [ x"$res" != x"$hash" ]; then
                               echo missmatch!  "$fname"
                               exit 1
                       fi
                       btmaketrackerless.py "$fname" &
               else
                       echo missing "$fname"
                       exit 1
               fi
       done < sha512.from.asc
    
       
  6. Once all the torrents have been generated, I then make the magnet links:
    
       $ cat btmakemagnet.sh
       #!/bin/sh -
    
       # metainfo file.: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso.torrent
       # info hash.....: 06091dabce1296d11d1758ffd071e7109a92934f
       # file name.....: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
       # file size.....: 203161600 (775 * 262144 + 0)
       # announce url..: udp://tracker.openbittorrent.com:80
       # btshowmetainfo 20030621 - decode BitTorrent metainfo files
    
       for i in *.torrent; do
               btshowmetainfo.py "$i" | awk '
       $0 ~ "^info hash" { info = $3 }
       $0 ~ "^file name" { name = $3 }
       END {
               print "magnet:?xt=urn:btih:" info "&dn=" name
       }'
       done
       
  7. I then create the magnet links file, and update the Torrents wiki page.

| Home |