Tuesday, July 3, 2018

Making FreeBSD magnet links

For the last few years, I've been producing torrents and publishing magnet links, but there is some special work that I do to make these.  The first few releases, I inserted a bogus tracker into the torrent, because despite there being plenty of tools out there for producing trackerless (DHT) torrents, they were all GUI and I never found any that were command line based.  The other was there was/is no tool for extracting the info hash and building the magnet link.  There may be tools now, but I couldn't find any when I started 3 years ago.

The following steps are based upon the recent release of FreeBSD 11.2-R, adjust as necessary.

  1. Fetch FreeBSD into a directory (I create a per release directory).  There are a few directories that you have mirror, I use wget for this.  The mirroring feature for wget isn't great.  After each command I have to remove the CHECKSUM.SHA256, CHECKSUM.SHA512 and index.html* files.
    $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/
    $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/aarch64/Latest/
    $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
    $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/i386/Latest/
  2. Fetch the signature files:
    $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,powerpc,powerpc-powerpc64,sparc64,arm64-aarch64}.asc
    $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,arm64-aarch64}-vm.asc
    $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-arm-armv6-{BANANAPI,BEAGLEBONE,CUBIEBOARD,CUBIEBOARD2,CUBBOX-HUMMINGBOARD,GUMSTIX,PANDABOARD,RPI-B,RPI2,WANDBOARD}.asc
  3. Verify the GPG key that signed the above files.  This is usually Glen Barber's key, but not always.  I have met and verified his fingerprint in person,  If you have verified someone's key who has signed Glen's key, that is another good way.
  4. Verify the checksum files:
    $ for i in *.asc; do gpg --verify $i; done
    You should see a bunch of lines like:
    Warning: using insecure memory!
    gpg: Signature made Fri Jun 22 09:33:50 2018 PDT
    gpg:                using RSA key 0x031458A5478FE293
    gpg: Good signature from "Glen Barber <gjb@FreeBSD.org>" [full]
    gpg:                 aka "Glen Barber <glen.j.barber@gmail.com>" [full]
    gpg:                 aka "Glen Barber <gjb@glenbarber.us>" [full]
    gpg:                 aka "Glen Barber <gjb@keybase.io>" [unknown]
    gpg: WARNING: not a detached signature; file 'CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-amd64-vm' was NOT verified!
    The last line can be ignored.  The non-.asc files were d/l'd and will not be used.  Make sure that all of the files report Good signature.
  5. In the past I have used BitTornado for other things, so I ended up using it as the basis to make the tool for creating trackerless torrent files.  The modifications were simple.  It appears that the original BitTornado CVS tree is off-line (anyways, it was served insecurely), but it looks like effigies/BitTornado is similar enough that it could be modified and used.  I copied btmakemetafile.py to btmaketrackerless.py and applied the following patch:
    $ diff -u btmakemetafile.py btmaketrackerless.py 
    --- btmakemetafile.py   2004-05-24 12:54:52.000000000 -0700
    +++ btmaketrackerless.py        2016-10-10 17:13:32.742081000 -0700
    @@ -23,9 +23,9 @@
     def prog(amount):
         print '%.1f%% complete\r' % (amount * 100),

    -if len(argv) < 3:
    +if len(argv) < 2:
         a,b = split(argv[0])
    -    print 'Usage: ' + b + ' <trackerurl> <file> [file...] [params...]'
    +    print 'Usage: ' + b + ' <file> [file...] [params...]'
         print
         print formatDefinitions(defaults, 80)
         print_announcelist_details()
    @@ -33,9 +33,9 @@
         exit(2)

     try:
    -    config, args = parseargs(argv[1:], defaults, 2, None)
    -    for file in args[1:]:
    -        make_meta_file(file, args[0], config, progress = prog)
    +    config, args = parseargs(argv[1:], defaults, 1, None)
    +    for file in args[0:]:
    +        make_meta_file(file, None, config, progress = prog)
     except ValueError, e:
         print 'error: ' + str(e)
         print 'run with no args for parameter explanations'

    If you notice, the only thing that is done is to drop the first argument, and instead of passing it into make_meta_file, a None is passed instead.  This will simply not add trackers to the torrent file.
  6. I then run the following script to verify the downloaded files, and generate the torrent files:
    $ cat cmp.sh 
    #!/bin/sh -
    # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/
    # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/aarch64/Latest/
    # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
    # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/i386/Latest/
    # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,powerpc,powerpc-powerpc64,sparc64,arm64-aarch64}.asc
    # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,arm64-aarch64}-vm.asc
    # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-arm-armv6-{BANANAPI,BEAGLEBONE,CUBIEBOARD,CUBIEBOARD2,CUBBOX-HUMMINGBOARD,GUMSTIX,PANDABOARD,RPI-B,RPI2,WANDBOARD}.asc

    grep -h '^SHA512' CHECK*.asc | sed -e 's/SHA512 (\(.*\)) = \(.*\)/\2 \1/' | sort -k 2 > sha512.from.asc

    while read hash fname; do
            if [ -e "$fname" ]; then
                    sigfile=`grep -l -- "$fname" *.asc | head -n 1`
                    echo checking "$fname", sig in: "$sigfile"
                    #res=`sha512 -q "$fname"`
                    res=`shasum -a 512 "$fname" | awk '{ print $1 }'`
                    echo "File is: $res"
                    if [ x"$res" != x"$hash" ]; then
                            echo missmatch!  "$fname"
                            exit 1
                    fi
                    if ! [ -e "$fname".torrent ]; then
                            btmaketrackerless.py "$fname"
                    fi
            else
                    echo missing "$fname"
                    exit 1
            fi
    done < sha512.from.asc
  7. Once all the torrents have been generated, I then make the magnet links:
    $ cat btmakemagnet.sh 
    #!/bin/sh -

    # metainfo file.: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso.torrent
    # info hash.....: 06091dabce1296d11d1758ffd071e7109a92934f
    # file name.....: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
    # file size.....: 203161600 (775 * 262144 + 0)
    # announce url..: udp://tracker.openbittorrent.com:80
    # btshowmetainfo 20030621 - decode BitTorrent metainfo files

    for i in *.torrent; do
            btshowmetainfo.py "$i" | awk '
    $0 ~ "^info hash" { info = $3 }
    $0 ~ "^file name" { name = $3 }
    END {
            print "magnet:?xt=urn:btih:" info "&dn=" name
    }'
    done
  8. I then create the magnet links file, and update the Torrents wiki page.
Sorry about the code formatting.  I don't know how to make it look better in blogger.

Friday, March 16, 2018

Unusable Insecurity

Many people claim that security is hard, and in many cases it is hard, but that isn't an excuse to make it harder than it needs to be. There are many layers to security, but adding extra layers, or making security controls inscrutable is a great way to ensure insecurity. Security needs to be simple and straightforward to configure, and easy to understand. There may be knobs for advanced users, but defaults need to be simple and correct.

I recently looked at using S3 as a shared store for some data. I was using the account New Context created for me that had limited AWS permissions. Creating the S3 bucket was simple enough, and making it not-public was too, but then I wanted to create a user/API key that only had access to the S3 bucket. Per Amazon IAM Best Practices, you should not share your account, but create new users for access. It turns out that I did not have the CreateUser permission. I involved a co-worker who did have permissions to create the user. Adding another person to the task makes things more complex through communication and their availability to work on it instead of their normal work.

As part of creating a user, you have to figure out what the Policy that you need to assign to the user. Amazon provides some Bucket Policy Examples, but none of them is a simple policy on granting read and write permissions to the bucket. There is an Amazon Policy Generator for helping you to create the policies, but it doesn't allow you to select buckets from your account (to simplify ARN [Amazon Resource Name] selection), and there are almost 70 actions provided in the selector. After some brief reading, I settled on a simple policy that I thought would allow the new user proper access: 4 permissions: PutObjects, GetObjects, ListObjects and RestoreObjects.

My co-worker created the user and applied the policy, but then I got an error handle code. Amazon does not provide an interface for turning on logging and/or querying why a request failed. Despite the error handle, I had ZERO insight into why the request failed. I could have involved AWS support, but now that would add yet another party in attempting to properly configure S3.

At this stage, I decided to give up, as I had already spent a few hours of my time, some of my co-worker's time, and a couple weeks due to various delays due to availability and other work. In this case, storing the data in S3 was more of a nicety, and I decided that checking the data into a private git repo was adequate compared to the complexities involved in configuring S3. git was a tried and tested way to store data and restrict access while S3 for this usage was not, and hard to configure.

After I wrote this blog post, a coworker linked me to the blog post titled Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket. It is concerning that this blog post has not been integrated, nor linked to from any of the IAM or S3 documentation. This is a valuable resource that should not be hidden.

I'm clearly not the only one that has had issues configuring S3 buckets. The end of 2017 has shown a large number of organizations fail to properly secure their S3 buckets, leaving many terabytes of data open for public download. It is unacceptable that such a service is so difficult to configure. The site https://s3stupidity.com/ lists the large number of breaches, many of which are by large companies who should have the technical chops (and $$) to properly configure it.

Security controls need to be simple and clear. Their descriptions need to be accurate and concise in what they do, and how they do it. Amazon does have a number of good resources, but they do not have a comprehensive guide for what each permission does. You cannot blame users for security failures when you make it next to impossible to configure properly.

Edited to remove a couple extra words.