encthenet ramblings

Simple Mobile Push Notifications

2024-09-17T00:00:00Z

Simple Mobile Push Notifications

Posted: September 17, 2024 at 12:00 PM

TL;DR: Check out WPN for code that implements push notifications to your browser (mobile or desktop).

Background¶

I run my own server/services, so it’s helpful to be able to receive notifications, such as when a hard drive starts throwing errors, or when I get an SMS to my home phone, or when snapaid fails to process a FreeBSD announcement. I had used a Java Signal client for a while, but that required both a phone number, but also a complex Java runtime to keep running, and eventually Signal upgraded the server protocol such that the client I had stopped working, and getting a new client working was too much work.

I was originally thinking about writing a custom Android App that just did this, but thinking about things a bit more, I realized that Web browsers have both a Push API and a Notifications API that I could use. Using an official notification API would keep the battery usage in check unlike using a custom protocol (or I’d have to use Google’s GCM directly, and yes, I do realize that web browsers are likely using it under the hood).

The blog post Push notifications overview has a good overview on how they work.

Overview¶

With a little bit of research, I found some example code that would register the push api, post the necessary info, and then display the notification received.

There are better guides on this, but the short is that it requests notifications, and once the user grants permission, registers a service worker, and once that service worker is registered, subscribe to the Push API, and POSTs the necessary info to the server. With the information POST‘d, there’s a simple client that can be used to push notifications.

With a little bit of work, I got it working and modified to be even more simple by not requiring a Python webserver, but instead use a CGI so that it integrates with your existing web server.

The installation and setup instructions are located in the WPN repository.

Isolated IP Management

2023-08-17T00:00:00Z

Isolated IP Management

Updated: August 25, 2023 at 4:15 PM

Originally Posted: August 17, 2023 at 12:00 PM

Edited on 2023-08-25 to add how to start the jail on each boot.

The latest iteration of my home firewall has a spare interface. Many switches these days have a dedicated management interface, and I wanted something similar for my firewall. I want an interface that I can plug in, get an IP via DHCP, and I can ssh into and it’ll “just work” even if there’s a misconfiguration, or an IP conflict.

On FreeBSD, vnet jails are a way to isolate an interface such that it won’t interfere (or receive interference) from other interfaces. Some may have just used epair and assigned IPs, but this could cause conflict, while using a unix domain socket keeps things isolated, and FreeBSD ships w/ everything needed to make this work. A package like socat isn’t needed.

Configuring the host was straight forward, make a directory for the socket:

mkdir /var/mgmt

And then add a line to /etc/inetd.conf to listen to incoming connections on the socket and launch sshd:

/var/mgmt/mgmt.ssh.sock stream  unix    nowait  root    /usr/sbin/sshd  sshd -i

The -i option to sshd tells it to run in “inetd” mode, which means that the stdin and stdout of the process is the socket to use for communication.

The next harder part was getting a jail configured that would accept incoming connections and forward them to the unix domain socket.

First the jail configuration, which goes in /etc/jail.conf or similar location:

mgmt {
        host.hostname = mgmt;                           # Hostname

        vnet ="new";
        vnet.interface="mgmt0";

        path = "/usr/jails/mgmt/root";                   # Path to the jail
        mount.fstab="/usr/jails/mgmt/fstab";             # mount spec

        mount.devfs;                                    # Mount devfs inside the jail
        devfs_ruleset = "101";

        exec.start = "/bin/sh /etc/rc";                 # Start command
        exec.stop = "/bin/sh /etc/rc.shutdown";         # Stop command
}

There isn’t anything special in this. It’s pretty standard jail configuration, the differences are the vnet configuration, and the devfs_ruleset.

The devfs_ruleset is necessary in order to expose the bpf interface used by dhclient. This required the following lines in /etc/devfs.rules:

[mydevfsrules_jail=100]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login

[devfsrules_jail_dhcp=101]
add include $mydevfsrules_jail
add path 'bpf*' unhide

Note that after adding the above lines, you need to run:

service devfs start

to load the rules (per devfs(8) man page).

Note: I learned my mistake not to number my blocks immediately after the standard defaults (from /etc/defaults/devfs.rules). I had done that befure but there’s now a conflict, so I skip ahead a bit to get a unique range.

Now I needed to make a number of directories for the jail:

mkdir -p /usr/jails/mgmt/root
mkdir -p /usr/jails/mgmt/etc
mkdir -p /usr/jails/mgmt/var/mgmt
mkdir -p /usr/jails/mgmt/tmp

I needed to setup the fstab for the jail:

# Device                Mountpoint      FStype  Options         Dump    Pass#
/                       /usr/jail/mgmt/root     nullfs  ro      0       0
/usr/jail/mgmt/etc      /usr/jail/mgmt/root/etc nullfs  rw      0       0
/usr/jail/mgmt/var      /usr/jail/mgmt/root/var nullfs  rw      0       0
/var/mgmt               /usr/jail/mgmt/root/var/mgmt    nullfs  ro      0       0
/usr/jail/mgmt/tmp      /usr/jail/mgmt/root/tmp nullfs  rw      0       0

This is a little bit more tricky, It first nullfs mounts the root system. I’m using ZFS boot environments, so this is a pretty clean FreeBSD install without much host specific data. It then mounts some jail specific directories for etc, tmp and var and finally mounts the shared directory w/ the unix domain socket to the host system. Also note that a couple of the mounts are read-only to prevent the jail from modifying the system.

The etc directory was populated from the system etc via:

tar -cf - -C /etc . | tar -xf - -C /usr/jails/mgmt/etc

Then the jail was configured, first /usr/jails/mgmt/etc/rc.conf:

hostname="mgmt.funkthat.com"

sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Management port
ifconfig_mgmt0="DHCP"

# necessary as devd can't be run in a jail
synchronous_dhclient="YES"

inetd_enable="YES"              # Run the network daemon dispatcher (YES/NO).

The key part of this configuration that took me a while to figure out was the synchronous_dhclient line. It used to be that netif would start dhclient, but in order to better handle USB ethernet devices and other removable interfaces, it was moved to devd. The only problem is that devd hasn’t been jail’ified, and you can’t run it to get things like link notifications that would normally launch dhclient. Setting this to yes, makes sure it gets launched when the jail starts.

And then the following line was added to /usr/jails/mgmt/etc/inetd.conf:

ssh     stream  tcp     nowait  root    /usr/bin/nc     nc -N -U /var/mgmt/mgmt.ssh.sock

This is the part that will forward incoming connections to the ssh port on to the unix domain socket.

Now that everything is configured, a simple jail -c mgmt will get the jail running and accepting connections.

To get the jail to start every boot, add the following to the host’s /etc/rc.conf:

jail_enable="YES"       # Set to NO to disable starting of any jails
jail_list="mgmt"

This was testing and deployed on a FreeBSD 14-CURRENT build as of August 8th, 2023, or more specifically, from main-n264621-09c20a29328, but it should work on all currently supported FreeBSD releases.

Making FreeBSD magnet links

2018-07-03T00:00:00Z

Making FreeBSD magnet links

Updated: December 7, 2022 at 12:20 PM

Originally Posted: July 3, 2018 at 4:49 PM

Note: This was originally posted here, but things have been improved since then and so it has been updated.

For the last few years, I’ve been producing torrents and publishing magnet links, but there is some special work that I do to make these. The first few releases, I inserted a bogus tracker into the torrent, because despite there being plenty of tools out there for producing trackerless (DHT) torrents, they were all GUI and I never found any that were command line based. The other was there was/is no tool for extracting the info hash and building the magnet link. There may be tools now, but I couldn’t find any when I started 3 years ago.

The following steps are based upon the recent release of FreeBSD 11.2‑R, adjust as necessary.

Fetch the FreeBSD release and release announcement into a directory (I create a per release directory). remove the CHECKSUM.SHA256, CHECKSUM.SHA512 and index.html* files.
```
   $ REL=12.4-RELEASE
   $ RELABV=12.4R
   $ curl https://www.funkthat.com/~jmg/FreeBSD-snap/snapshot.idx.xz | xzcat | awk ' $2 == "'"$REL"'" { print $9 }' | xargs -L 1 -P 3  wget --no-verbose -c --limit-rate=7000k
   $ wget https://www.freebsd.org/releases/$RELABV/announce.asc

   
```
This step depends upon snapaid running. If it is no longer running for some reason, addinfo.sh could be run manually against the release email (not the downloaded announce.asc file as the message id is needed) to generate snapshot.idx.xz locally, and use that instead.
Verify the GPG key that signed the above files. This is usually Glen Barber’s key, but not always. I have met and verified his fingerprint in person, If you have verified someone’s key who has signed Glen’s key, that is another good way.

Verify the release announcement:


   $ gpg --verify announce.asc
   Warning: using insecure memory!
   gpg: Signature made Mon Dec  5 17:34:59 2022 PST using RSA key ID 478FE293
   gpg: Good signature from "Glen Barber <gjb @ FreeBSD.org>" [unknown]
   gpg:                 aka "Glen Barber <gjb @ keybase.io>" [unknown]
   gpg:                 aka "Glen Barber <gjb @ glenbarber.us>" [unknown]
   gpg:                 aka "Glen Barber <glen.j.barber @ gmail.com>" [unknown]
   gpg: WARNING: This key is not certified with a trusted signature!
   gpg:          There is no indication that the signature belongs to the owner.
   Primary key fingerprint: 78B3 42BA 26C7 B2AC 681E  A7BE 524F 0C37 A0B9 46A3
        Subkey fingerprint: 8D12 403C 2E6C AB08 6CF6  4DA3 0314 58A5 478F E293

In the past I have used BitTornado for other things, so I ended up using it as the basis to make the tool for creating trackerless torrent files. The modifications were simple. It appears that the original BitTornado CVS tree is off-line (anyways, it was served insecurely), but it looks like effigies/BitTornado is similar enough that it could be modified and used. I copied btmakemetafile.py to btmaketrackerless.py and applied the following patch:


   $ diff -u btmakemetafile.py btmaketrackerless.py 
   --- btmakemetafile.py   2004-05-24 12:54:52.000000000 -0700
   +++ btmaketrackerless.py        2016-10-10 17:13:32.742081000 -0700
   @@ -23,9 +23,9 @@
    def prog(amount):
        print '%.1f%% complete\r' % (amount * 100),

   -if len(argv) < 3:
   +if len(argv) < 2:
        a,b = split(argv[0])
   -    print 'Usage: ' + b + ' <trackerurl> <file> [file...] [params...]'
   +    print 'Usage: ' + b + ' <file> [file...] [params...]'
        print
        print formatDefinitions(defaults, 80)
        print_announcelist_details()
   @@ -33,9 +33,9 @@
        exit(2)

    try:
   -    config, args = parseargs(argv[1:], defaults, 2, None)
   -    for file in args[1:]:
   -        make_meta_file(file, args[0], config, progress = prog)
   +    config, args = parseargs(argv[1:], defaults, 1, None)
   +    for file in args[0:]:
   +        make_meta_file(file, None, config, progress = prog)
    except ValueError, e:
        print 'error: ' + str(e)
        print 'run with no args for parameter explanations'

If you notice, the only thing that is done is to drop the first argument, and instead of passing it into make_meta_file, a None is passed instead. This will simply not add trackers to the torrent file.

I then run the following script to verify the downloaded files, and generate the torrent files:


   $ cat cmp.sh
   #!/bin/sh -
   # REL=12.4-RELEASE
   # RELABV=12.4R
   # xzcat ~jmg/public_html/FreeBSD-snap/snapshot.idx.xz | awk ' $2 == "'"$REL"'" { print $9 }' | xargs -L 1 -P 3  wget --no-verbose -c --limit-rate=7000k
   # XXX - following command 404, manually saved from relase announcement
   # wget https://www.freebsd.org/releases/$RELABV/announce.asc
   # wget -c https://download.freebsd.org/ftp/releases/CI-IMAGES/$REL/amd64/Latest/FreeBSD-$REL-amd64-BASIC-CI.raw.xz
   # wget -c https://download.freebsd.org/ftp/releases/VM-IMAGES/$REL/riscv64/Latest/FreeBSD-$REL-riscv-riscv64.{qcow2,raw,vhd,vmdk}.xz

   grep -h '^   SHA512' announce.asc | sed -e 's/SHA512 (\(.*\)) = \(.*\)/\2 \1/' | sort -k 2 > sha512.from.asc

   while read hash fname; do
           if [ -e "$fname" ]; then
                   if [ -e "$fname".torrent ]; then
                           echo skipping "$fname"...
                           continue
                   fi

                   sigfile=$(grep -l -- "$fname" *.asc | head -n 1)
                   echo checking "$fname", sig in: "$sigfile"
                   #res=`sha512 -q "$fname"`
                   #res=`shasum -a 512 "$fname" | awk '{ print $1 }'`
                   res=$(openssl sha512  < "$fname" | awk '{ print $2 }')
                   echo "File is: $res"
                   if [ x"$res" != x"$hash" ]; then
                           echo missmatch!  "$fname"
                           exit 1
                   fi
                   btmaketrackerless.py "$fname" &
           else
                   echo missing "$fname"
                   exit 1
           fi
   done < sha512.from.asc

Once all the torrents have been generated, I then make the magnet links:


   $ cat btmakemagnet.sh
   #!/bin/sh -

   # metainfo file.: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso.torrent
   # info hash.....: 06091dabce1296d11d1758ffd071e7109a92934f
   # file name.....: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
   # file size.....: 203161600 (775 * 262144 + 0)
   # announce url..: udp://tracker.openbittorrent.com:80
   # btshowmetainfo 20030621 - decode BitTorrent metainfo files

   for i in *.torrent; do
           btshowmetainfo.py "$i" | awk '
   $0 ~ "^info hash" { info = $3 }
   $0 ~ "^file name" { name = $3 }
   END {
           print "magnet:?xt=urn:btih:" info "&dn=" name
   }'
   done

I then create the magnet links file, and update the Torrents wiki page.

tmux beginners guide

2022-11-15T00:00:00Z

tmux beginners guide

Posted: November 15, 2022 at 12:00 PM

tweets

Originally Posted: Sun Aug 07 18:40:30 +0000 2022

First, here’s the .tmux.conf I use (alt text to copy/paste).

This remaps the control key from ctrl-b (^B) to ctrl-a which is more natural for me.

Next is split windows, to have two windows side by side: ^A %, to have top/bottom: ^A “.

To move between windows, ^A then one of h (up) j (left) k (right) l (ell down), aka the vi movement keys, or you can use arrow keys.

To resize the current window, ^A < or > or + or -.

To swap two windows (to move them around), ^A { or ^A }.

To copy between windows, ^A [. Then move the cursor to where you want to start, hit space to start selection, move to the end, hit enter. Now that you have your selection, to paste it, ^A ].

To create another workspace in the same session ^A c. To switch workspaces, ^A num.

Note that my window/workspace terminology doesn’t line up w/ tmux’s, but makes more sense to me.

fin.

Oh, another thing. You have have multiple terminals connected to the same session (tmux attach). This is great for pair programming, so you both don’t have to crowd around a terminal and swap keyboards, each can be on their own laptop, or now w/ WFH, be far away.

Nearly Complete Guide to RNG on a microcontroller

2022-02-12T00:00:00Z

Nearly Complete Guide to RNG on a microcontroller

Posted: February 12, 2022 at 11:50 AM

Security depends upon cryptography and which in turn depends upon a Random Number Generator (RNG). An RNG is used for key generation (both symmetric and asymmetric) and key negotiation (session establishment). The later is an absolute requirement to ensure that communications can be secured. The former (key generation) can be used at first boot for personalization, but isn’t necessary as it could be done when personalizing the device at programming or first deployment.

There are two types of RNGs, the first is a True Random Number Generator (TRNG). This is one that takes some non-deterministic process, often physical, and measures it. Often, these are slow and are not uniform, requiring a post processing step before the are useful.

The second type is a Pseudo Random Number Generator (PRNG)NIST also refers to a PRNG as a Deterministic Random Bit Generator (DRBG).. PRNGs take a seed, and can generate large, effectively unlimited amounts of random data, when seeded properly. The issue is than if someone is able to obtain the seed, they will be able to predict the subsequent values, allowing breaking security.

The standard practice is to gather data from a TRNG, and use it to seed a PRNG. It used to be common that the PRNG should more additional random data mixed in, but I agree w/ djb (D. J. Bernstein) that once seeded, no additional seeding is neededSee his blog post Entropy Attacks! as modern PRNGs are secure and can generate random data such that their state will not leak.That is, taking it’s output, that neither past nor future output can be predicted.

There are lots of libraries and papers that talk about how to solve the problem for RNGs on a microcontroller that may not have an integrated [T]RNG block, but I have not been able to find a complete guide for integrating their work into a project where even a relative beginner could get it functional.

This article was written as I developed the lora-irrigation project. This project will be used as an example, and the code reference is mostly licensed under the 2-clause BSD license, and so is freely usable for your own projects.

Sources of Randomness¶

As mentioned, most microcontrollers do not have a dedicated hardware block like modern AMD64 (aka x86-64) processors do w/ the RDRAND instruction. Though they do not, there are other sources that are available.

The first, and easiest one is the Analog Digital Converter (ADC). Even if the ADC pin is tied to ground, the process of digital conversion is not 100% deterministic as there are errors in the converter or noise introduced on the pin.The article ADC Input Noise: The Good, The Bad, and The Ugly. Is No Noise Good Noise? talks about this.

The data sheet for the microcontroller will help determine the expected randomness from the part. In the case of the STM32L151CC that I’m using, Table 57 of the data sheet lists the Effective number of bits (ENOB) as typically 10 bits, which is a couple bits short of the 12 bit resolution of the ADC. This means that the 2 least significant bits are likely to have some noise in them. I did a run, and collected 114200 samples from the ADC. The Shannon entropy calculated using the empirical probabilities was 2.48. Now this is not strictly Shannon entropy, as the values were calculated from the experiment, and Shannon entropy should be calculated from the a priori probabilities. Discarding the 0’s (which makes up over half the results) improves the entropy calculation to 3.29. The min-entropy, Forward reference: min-entropy awk script a better indicator of entropy, calculation is 1.2 bits, and if all the 0’s are dropped, it improves to 2.943. This does help, but in the end, subtracting the data sheet’s ENOB from the ADC resolution does result in an approximate estimate of entropy.

It is possibly that a correlation analysis between samples could further reduce the entropy gathers via the ADC, but with sufficient collection, this should be able to be avoided.

The second is using uninitialized SRAM. It turns out that this has been studied in Software Only, Extremely Compact, Keccak-based Secure PRNG on ARM Cortex-M and Secure PRNG Seeding on Commercial Off-the-Shelf Microcontrollers. Depending upon how the SRAM is designed in the chip, it can create a situation where each bit of SRAM will be indeterminate at boot up. Both of these papers studied a similar microcontroller, an STM32F100R8 to the one I am using, a STM32L151CC.

I ran my own experiments where I powered on an STM3L151CC and dumped the SRAM 8 times and analyzed the results. I limited my analysis to 26863 bytes the 32 KiBytes of ram (remaining was data/bss or stack, so would not change, or was zeros). I then calculated the min-entropy for each bit across power cycles and the resulting sum was 11188, or approximately .416 bits per byte. This is 5.2% and in line with what the later paper observed for a similar device.

Part of using a source of randomness is making sure that it is usable. In the case of the ADC, each reading can be evaluated against previous reads to ensure that the data being obtained is possibly random. In the case of SRAM, this is more tricky, as the state of SRAM is static, and short of a reset, will not change. This means that to use SRAM, proper analysis of the device, or family of devices, need to be evaluated for suitability. There are cases where a device’s SRAM does not provide adequate entropy, as discussed in the papers, and so this method should not be used in those cases, or not solely relied upon.

The following is an awk script for calculating the min-entropy of the provided data. Each sample must be the first item on a line, and each sample must be a hexadecimal value w/o any leading 0x or other leading identifier:

# Copyright 2021 John-Mark Gurney
# This script is licensed under the 2-clause BSD license

function max(a, b)
{
        if (a > b)
                return a;
        else
                return b;
}

{
        v = ("0x" $1) + 0; a[NR] = v;
        maxv = max(maxv, v);
}

END {
        tcnt = length(a);
        me = 0;
        for (bit = 0; 2^bit <= maxv; bit += 1) {
                cnt0 = 0;
                cnt1 = 0;
                for (i in a) {
                        tbit = int((a[i] / 2 ^ bit) % 2);
                        if (tbit)
                                cnt1 += 1;
                        else
                                cnt0 += 1;
                }
                v = -log(max(cnt0, cnt1) / tcnt) / log(2);
                print "bit " bit ":\t" v;
                me += v;
        }
        printf "total:\t%0.3f\n", me;
}

It is also possible that there are other parts of the board/design that could be a source of randomness. The project that started this journey is using LoRa for communication. It turns out that the sample code for the radio chip (LoRaMac‑node) implements a random interface. The function just waits one milisecond, reads the RSSI value, takes the low bit and repeats this 32 times to return a 32-bit word. There are issues with this as I cannot find any description of the expected randomness in the data sheet, nor in the code. It also does not do any conditioning, so just because it returns 32-bits, does not guarantee 32-bits of usable entropy. I have briefly looked at the output, and there does appear to be higher lengths of runs than expected. Another issue is that it’s collection takes a while, as the fastest is 1 bit per ms. So, assuming the need to collect 8 bits for 1 bit of entropy (pure speculation), that means at minimum 2 seconds to collect the 2048 bits necessary for 256 bits of entropy.

Uniquifying¶

One of the other ways to help ensure that a microcontroller is to integrate per device values into the PRNG. This does not guarantee uniqueness between boots, but it does make it harder to attack if an attacker is able to control the other sources of randomness.

In the case of the STM32L151 chip I am using, there is a unique device id register. The device register is programmed at the factory. Because it is unknown if this unique id is recorded by the manufacturer, and possibly traced through the supply chain, and no guarantees are made to both the uniqueness or privacy, it has limited use to provide any serious additional randomization.

Another method, is to write entropy at provisioning time. This can be done in either flash memory or EEPROM, which may have a more granular write access.

Using SRAM¶

The tricky part of using SRAM is figuring out how to access the uninitialized memory. Despite having full access to the environment, modifying the startup code, which is often written in assembly, to do the harvesting makes an implementation less portable. Using standard C, or another high level language, makes this easier, but we need to know where the end of the data and bss segments are. This is where looking at the linker script will come in.

A linker script is used to allocate and map the program’s data to the correct locations. This includes allocating memory so that all the code and data fits in flash, but also allocating ram for variables, and stack. Often there will be a symbol provided that marks where the data and bss sections in ram end, and the heap should begin. For example, in STM32L151CCUX_FLASH.ld at lines 185 & 186 it defines the symbols end and _end, the later of which is often used by sbrk (or _sbrk in my project’s case in libnosys A sample _sbrk is in utils_syscalls.c, though this particular implementation is not used by my project.) to allocate memory for the heap. Using sbrk is the easiest method to access uninitalized SRAM, but modifying or adding a symbol can be used if your microcontroller’s framework does not support sbrk.

Putting it together¶

It is accepted that integrating as many difference sournces of entropy (TRNGs) is best. This ensures that as long as any single soruce is good, or each one is not great, but combined they provide enough entropy (preferably at least 128 bits), that the seeded PRNG will be secure and unpredictable.

As some sources are only available at first boot, e.g. SRAM, it is best to save a fork of the PRNG to stable storage. In my implementation, I decided to use EEPROM for this. I added an additional EEPROM section in the linker script, and then added a symbol rng_save that is put in this section. This should be 256-bits (32-bytes) as the savings of smaller does not make sense, and any proper PRNG when seeded with 256-bits will provide enough randomness. Writing to EEPROM does require a little more work to have the code save to this region, rather than RAM, but the STM32 HAL layer has functions that make this easy.

It would be great if the PRNG seed could be stored in read-once, write-once memory to ensure that it can be read, mixed in with any additional entropy, and then written out, but I do not know of any microcontroller that supports this feature.

Part of this is is to ensure that the the state between the saved seed, and the PRNG state used for this boot is disjoint, and that if either seed is compromised, neither can be backtracked to obtain the other. In the case of strobe, the function strobe_randomize does a RATCHET operation at the end, which ensure the state cannot be rolled back to figure out what was generated, and as the generated bytes does not contain the entire state of the PRNG, it cannot be used to reconstruct the future seed.

Another advantage of using EEPROM is the ability to provide an initial set of entropy bytes at firmware flashing time. I did attempt to add this, but OpenOCD, which I use for programming the Node151 device, does not support programming EEPROM, so in my case, this was not possibleDespite not using it, the infrastructure to generate perso entropy is still present in the Makefile.. I could have added an additional source data file to the flash, but figured that the other sources of entropy were adequate enough for my project.

CODEpendence

2021-07-07T00:00:00Z

CODEpendence

Posted: July 7, 2021 at 11:16 AM

TL;dr: If you use submodules that point to a GitHub repo, make sure that the commit id matches an offical branch or tag, especially if upgraded via a PR or submitted patch.

This issue was disclosed to GitHub via the HackerOne Bug Bounty program and resolved by them in a timely manner. The writeup is available and is the same one that was provided to GitHub. It contains the complete steps in more detail than this blog post does.

Discovery¶

Earlier this year, I was dealing with a git repo that used submodules. I’ve never been a fan of them due to the extra work involved in using them. But then a thought hit me, last year, when GitHub took down youtube-dl, someone was a bit sneaky and inserted a copy of it into GitHub’s DMCA repo. They were able to do this because there is a feature/bug in GitHub’s backend, that all the commits to a forked repo are accessible in the parent repo, it’s just that the branches and tags are maintained separately. This makes sense to reduce storing duplicate data if a repo is large or has many forks.

Verification¶

The next question, would combining these into an attack even work? What would things look like? I created a few accounts to test them, creating a project to represent a code dependancy, depproj, that would be imported into another project by another user, proj. Then once those were created, have a malicious user create a fork of both the deprpoj and the proj.

Once the malicious forks were created, clone them locally. With the clones, malicious code can be inserted into the depproj repo. If you look at the repo, the previous commit was done as the maliciousrepo user, but while I was working on this, I remembered that w/ git, you can set the commit author to be anything (signing helps prevent that), so this commit appears to be done by the correct upstream123 user.

Once the malicious code has been inserted, the malicious user can now update the submodule of the project to the commit id of the malicious code. This is done simply by doing:

cd depproj
git fetch origin <commitid>
git checkout <commitid>

Even though the depproj still points to the upstream123 repo, because fork commits appear IN the depproj repo, the above works w/o any other changes. This is also what makes it dangerous, because the repo is not changed, it can be disguised as a simple version update.

A PR is then submitted to the project being attacked. I did not control the author of commits as well as I should have, but it still is effective. If you click into the proposed change, and then click on code.c, the file changed, it’ll bring you to the change compare view. For this demo, it was a small change, but if the project is large, it’s would be easy to bury a minor flaw in lots of changes. The other thing to note about this page is that the author displayed is NOT the author of the change, but it appears that it is a legitimate change by the author of the repo.

Conclusion¶

This is an interesting attack in that it leverages two features in a way that has surprising results. It demonstrates that software dependancies need to be reviewed, and vetted, and that if you’re using GitHub, that just because a PR says it’s updating a submodule to the new version, it doesn’t mean that it is safe to simply merge in the change.

Timeline¶

2021-03-31 — Reported to GitHub via HackerOne.
2021-03-31 — More info requested and provided.
2021-04-01 — Ack’d issue and started work on fix.
2021-05-04 — GitHub determined it was low risk, but did add warning when viewing commit.
2021-05-05 — Asked GitHub for disclosure timeline.
2021-06-04 — Pinged GitHub again.
2021-07-07 — Published blog post.

Installing FreeBSD on a PC Engines APU4C2

2019-08-08T00:00:00Z

Installing FreeBSD on a PC Engines APU4C2

Posted: August 8, 2019 at 11:08 PM

FreeBSD

I recently upgraded my internet connection, and needed some additional performance, so I purchased a APU.4C2. When I tried to boot it up, I would get to loading the kernel, but then I would not get anything.

After a quick search I found Dr. David A. Eckhardt’s post on How to Install FreeBSD 12.0 on a PC Engines apu2 Machine (apu4c4). The instructions require mounting the memstick image on a FreeBSD box, but I don’t have one hand for mounting the image. Once I knew what I needed to do, I realized I didn’t need to mount the image to set the options needed.

First on boot, I hit F10 so that I’d be presented with a boot menu. Strictly this isn’t needed, but helped me time the next part. I then selected the USB drive that had the memstick image, and as soon as I selected the image, I immediately started hitting space bar. This dropped me into the first stage boot loader:

Select boot device:

1. USB MSC Drive Generic STORAGE DEVICE 0566
2. ata0-0: SATA SSD ATA-11 Hard-Disk (15272 MiBytes)
3. Payload [memtest]
4. Payload [setup]

Booting from Hard Disk...
-
FreeBSD/x86 boot
Default: 0:ad(0,a)/boot/loader
boot:

This is the point to enter in the line that would normally go into boot.conf: -h -S115200 -v

Once that is entered, this proceeded to the loader boot loader. I selected 3 to escape to the loader prompt. This allowed me to enter some of the commands that would go into loader.conf. Note that the commands w/ a . in them need to be preceded w/ the set command:

Type '?' for a list of commands, 'help' for more detailed help.
OK boot_serial="YES"
OK console="comconsole"
OK comconsole_speed="115200" 
OK set kern.cam.boot_delay="15000"
OK set kern.cam.scsi_delay="15000"
OK set hw.igb.enable_msix=0
OK set hw.pci.enable_msix=0
OK set hint.ahci.0.msi="0"
OK set hint.ahci.1.msi="0"
OK boot

And as you can see, I finally booted the kernel using “boot“.

This started the install process and things proceeded as normal. I decided to use GPT instead of MBR for the partitioning scheme. It is not a big deal which one is picked. I also did the recommended settings so that upon reboot the above settings would keep. I do plan on attempting to flash the bios and see if I can’t get MSI and/or MSI-X working, as that would be a huge advantage for interrupts.

One option for reducing CPU heat is to run powerd. The powerd daemon will monitor CPU usage, and dynamically adjust the frequency of the CPU to match the current load.

Using Signal on a server

2019-04-08T00:00:00Z

Using Signal on a server

Posted: April 8, 2019 at 1:54 PM

For a long while, I’d been using an email to SMS gateway to push important notifications from my server, such as SMART error messages, to my phone. After all the NSA warrantless surveillance, I made a commitment to encrypt as much of my communications as possible. When Signal came out, I adopted it because of it’s strong encryption and privacy. Ever since I’ve been wanting to use it for notifications from my server. I finally got around to trying out the CLI version, and got it to work.

The installation of the command line utility for Signal was more straight forward than I was expecting. I decided to use signal-cli and I was a bit worried, as it uses Java. Java has historically been difficult to run on FreeBSD due to lack of support and draconian licensing terms. I was surprised that the packages for OpenJDK 8 were both present and just worked on my server. A simple pkg install openjdk8 got Java up and running.

One thing to note is that the package said that fdesc and proc needed to be mounted for Java to work, but I did not, and things still worked. There are likely other parts of Java that may not work w/o those mounted, but not for Signal.

As I have been using OSS for a long time, I like to build things from source, so I followed the instructions at Building signal-cli and got the command built with out any trouble.

Once the command was built, the Usage guide provided the basics, but didn’t include instructions on how to verify the safety numbers to ensure that the initial exchange was not MitM’d. There is a man page, but it requires a2x and separate steps to build, but a little bit of digging got me the necessary steps (also, it turns out that the adoc format is a simple text format).

With a bit of searching, I found the listIdentities and verify commands. There may have been another way, but because I had sent a test message, my phone was listed:

$ signal-cli -u +XXXXXXXXXXX listIdentities
+YYYYYYYYYYY: TRUSTED_UNVERIFIED Added: Sat Apr 06 18:43:15 PDT 2019 Fingerprint: ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ  Safety Number: WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW

And then I needed to use the trust subcommand:

$ signal-cli -u +XXXXXXXXXXX trust -v 'WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW WWWWW' +YYYYYYYYYYY

The hardest part of this was figuring out how to invoke the command upon reception of an email. I used an alias listed in /etc/aliases to forward the email to both the SMS gateway and myself. The issue with trying to invoke the command from here was that the command was run as the mailnull user, which of course didn’t have access to my user’s home directory to read the private key. After a bit of debating, I remembered I use procmail, and realized this was the best way to send the message.

I created a symlink for the command into my user’s bin directory, created a short script called sendcell:

$ cat ~/bin/sendcell
#!/bin/sh -

~user/bin/signal-cli -u +XXXXXXXXXXX send +YYYYYYYYYYY

and then added a filter to my .procmailrc file. The filter at first looked like this:

:0Wf
* ^TO_celluser@([^@\.]*\.)*example.com
| sendcell

But after the first test, it included all the headers, including all the Received headers, so I updated it to use formail to remove all but the From, Subject and Date (in case the message gets significantly delayed, I can see by how much) headers:

:0c
* ^TO_celluser@([^@\.]*\.)*example.com
{
:0Wf
| formail -k -X From: -X Subject: -X Date:

:0
| sendcell
}

and now I get the messages delivered to my phone securely!

It is tempting to use this to be able to invoke commands on my server remotely, but there isn’t much I need to do when I don’t have my laptop with me.

Crash Dumps: Do I submit them?

2018-10-23T00:00:00Z

Crash Dumps: Do I submit them?

Posted: October 23, 2018 at 3:54 PM

security

TL;DR: No, do not submit your crash dumps. Consumers: No company has sane crash dump policies to ensure your privacy and PII is protected, minimized and secured. Companies: You need to ensure that crash dumps are handled in a secure manner and that crash dumps are just that: a crash dump. Anything not directly related to a crash dump should be excluded. Usage statistics and the like do not belong in crash reports.

Why Not Send Dumps?¶

There is a long history of companies failing to minimize the data and to protect it. Microsoft for years sent crash dumps over the internet in the clear (WER & Privacy conerns). This allowed the NSA to harvest them, and develop 0-days for issues that MS failed to fix. Google’s Chrome would send a screencap of the entire Desktop along with it’s crash dumps (link). It previously would only send the window, but now sends the entire screen. Though they provide a preview, there is no way to see exactly what information will be sent.

I do not relish in advising people to not submit crash dumps as this will impact developers ability to fix bugs. But as with all aspects of security, companies continue to demonstrate that they are not willing to do the work that is necessary to protect user’s data and their privacy.

Communication¶

You need to communicate to your users how crash dumps are handled. Just saying, trust us, does not inspire confidence, as there are a large number of cases of data breaches where the company has said exactly that leading up to leaks. The policy is the first step to demonstrating that you have thought about user’s concerns and decided how you will handle their personal and sensitive data.

The policy also helps shape how employees will treat the data too. By having the policy, it is a reiteration to the employees that user data isn’t simply chaff, but that it needs to be protected and handled with care.

Just saying that it’s protected by a privacy policy isn’t enough. For example, Google Chrome’s Report an Issue says that the information is protected by their privacy policy, but if you read the Chrome browser Privacy Policy, there is nothing in there that says how the data is handled. That it is handled like the rest of the data collected does not inspire confidence that the possibly confidential data that may be included will be handled with greater care.

How to handle dumps¶

The first step is to ensure that what is collected in the dump has minimum information needed to debug issues. Code paths (back traces) are likely to be safe. Data, such as arguments to functions, may include user data and needs to be carefully examined. There are many different types of data that can be released from embarrassing (what website was visited), to security breach (including cookies/tokens for web sites that may not be yours), to confidential intellectual property leaking (source code, designs, etc). Each of these may have different impact on the user, but should never happen.

Second, crash dumps need to be transmitted confidentially. This means either using TLS or encrypting the dumps with a tool like GPG before sending. This ensures that unauthorized parties are unable to view the contents. The NSA used the dumps to gather information for their operations, which if Microsoft had properly protected their user’s data, this would not have happened.

Third, they need to be stored in a secure manner and able to be expunged. It should even be possible for the user to remove the crash dump if they discover that information was shared when it should not have been. The life time that a company keeps the dumps should be limited. If you haven’t fixed a bug from five years ago, how do you know you can reproduce it, or that if you are able to reproduce it, that the code is still present in your current software? It the crash is a major issue, it is likely that you’ll have more recent dumps that exhibit the same issue if it is a problem, so old dumps are just not as useful compared to the data that may be present.

As crash data needs to be deleted, almost any cloud service is immediately excluded unless other precautions are used, such as encryption. With the cloud, you have zero visibility into how the data is managed and how or when it is backed up. Cloud providers rarely tell you their retention policies on back ups, and other policies that may keep data around. Do they securely remove your VM’s storage when they migrate it? Do they ensure that storage is deleted from all clones, shards, servers and backups when you delete it? If not, how long will that data stay around before it is finally expunged.

Fourth, access to dumps need to be controlled. Auditing is a good first step to know who is accessing the data, but additional measures like limiting who has access needs to be used. Not everyone on the team needs access to them. As they are classified, they can be assigned to teams or people that need access to the data in them. This helps make sure that an employee isn’t trolling for nudes or other confidential information. It should also limit how easy data is copied out of the archive. How these controls are put in place will vary by company.

Edit: Case in point: I recently opened a support case with Apple. Apple provides a program to collect data to send to them to help trouble shoot the issue. The program collected 280 MB of data. When uploading the data, Apple informs the user that it is their responsibility to NOT submit any personal information that they don’t want. There is no way most people are qualified to look at the data, and even redact it properly. I attempted to do so, and it took a very long time, and I’m not sure that I got everything. Expecting a normal computer user to be able to do this is insane.

TLS Client Authentication Leaks User Info (pre-TLS1.3)

2018-10-15T00:00:00Z

TLS Client Authentication Leaks User Info (pre-TLS1.3)

Posted: October 15, 2018 at 10:54 AM

security
TLS

It’s been long known that TLS is not the best privacy protecting protocol in that SNI leaks what domain the client connects to. I’m a bit surprised that I haven’t seen the failure to protect user information when using client authentication mentioned, but it’s likely that TLS client authentication is so rarely used, that this have not been on anyone’s radar.

TL;DR: Just don’t use TLS client authentication on anything before TLS 1.3.

With TLS 1.2 and earlier, if you use client authentication, the client certificate is transmitted in the clear. This contains enough information to uniquely identify the user. If it didn’t, then there would be no way for the server to do the authentication.

The danger of this is that Eve (eavesdroppers) on path will be able to track your user’s (or your) connections, where they connect from, figure out how much data they transfer between to/from your site and likely profile their usage.

I was confident that this was the case as I know that the entire handshake is in the clear. It isn’t till the Finished messages that the session becomes encrypted. (TLS 1.3 fixed this by using a new derived key, [sender]_handshake_traffic_secret, to encrypt all the server params, which the client will use to encrypt it’s response to the certificate request in the server params.) I decided to verify that this was the case.

I generated a server and a client certificate and key:

openssl req -batch -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout server.key -out server.crt
openssl req -batch -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout client.key -out client.crt

I then launched the server, and included the -Verify and -CAfile options for s_server to request a client certificate:

openssl s_server -accept 5829 -cert server.crt -key server.key -Verify 5 -CAfile client.crt -debug

Then I ran tcpdump to capture the session:

sudo tcpdump -s 0 -n -i lo0 -w clientcert.tcpdump port 5829

And then the client to connect to the server:

openssl s_client -connect localhost:5829 -key client.key -cert client.crt -debug

A usual, non-client authenticated connection and close was about 17 packets, but when I included the client authentication, it became 42 packets (the answer!).

I loaded the packet capture into wireshark, applied the SSL protocol analysis and confirmed that the client certificate was present in clear text:

So, there you have it. Do not use client authentication, pre-TLS 1.3, if you care about the privacy of your users.

It is safe to use client authentication w/ a TLS 1.3 server as long as the server requires all clients be 1.3 clients. If the key exchange algorithm is one of DHE_DSA, DHE_RSA, or an ECDH key exchange algorithm, the random bytes in the Hello messages are signed and these bytes are used by TLS 1.3 for downgrade protection. As the signature covers these bytes, the client would be able to detect any attempts to modify the server or client handshake messages to force a downgrade before it would send the client certificate.

Thanks to Mike Hamburg for reviewing an earlier version of this blog post and pointing out that TLS 1.3 was not vulnerable to this and helping w/ some of the research to prove it.

References:

TLS 1.2 Protocol Diagram
TLS 1.2 ServerKeyExchange
ECC Cipher Suites for TLS ServerKeyExchange Signature
TLS 1.3 Protocol Diagram
TLS 1.3 Server Hello Downgrade Protection

Making FreeBSD magnet links

2018-07-03T00:00:00Z

Making FreeBSD magnet links

Posted: July 3, 2018 at 4:49 PM

⚠️ This post is old, the most up to date one is here.

The following steps are based upon the recent release of FreeBSD 11.2‑R, adjust as necessary.

Fetch FreeBSD into a directory (I create a per release directory). There are a few directories that you have mirror, I use wget for this. The mirroring feature for wget isn’t great. After each command I have to remove the CHECKSUM.SHA256, CHECKSUM.SHA512 and index.html* files.


   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/aarch64/Latest/
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
   $ wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/i386/Latest/

Fetch the signature files:


   $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,powerpc,powerpc-powerpc64,sparc64,arm64-aarch64}.asc
   $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,arm64-aarch64}-vm.asc
   $ wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-arm-armv6-{BANANAPI,BEAGLEBONE,CUBIEBOARD,CUBIEBOARD2,CUBBOX-HUMMINGBOARD,GUMSTIX,PANDABOARD,RPI-B,RPI2,WANDBOARD}.asc

Verify the GPG key that signed the above files. This is usually Glen Barber’s key, but not always. I have met and verified his fingerprint in person, If you have verified someone’s key who has signed Glen’s key, that is another good way.

Verify the checksum files:


   $ for i in *.asc; do gpg --verify $i; done
   You should see a bunch of lines like:
   Warning: using insecure memory!
   gpg: Signature made Fri Jun 22 09:33:50 2018 PDT
   gpg:                using RSA key 0x031458A5478FE293
   gpg: Good signature from "Glen Barber <gjb@FreeBSD.org>" [full]
   gpg:                 aka "Glen Barber <glen.j.barber@gmail.com>" [full]
   gpg:                 aka "Glen Barber <gjb@glenbarber.us>" [full]
   gpg:                 aka "Glen Barber <gjb@keybase.io>" [unknown]
   gpg: WARNING: not a detached signature; file 'CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-amd64-vm' was NOT verified!

The last line can be ignored. The non-.asc files were d/l’d and will not be used. Make sure that all of the files report Good signature.


   $ diff -u btmakemetafile.py btmaketrackerless.py 
   --- btmakemetafile.py   2004-05-24 12:54:52.000000000 -0700
   +++ btmaketrackerless.py        2016-10-10 17:13:32.742081000 -0700
   @@ -23,9 +23,9 @@
    def prog(amount):
        print '%.1f%% complete\r' % (amount * 100),

   -if len(argv) < 3:
   +if len(argv) < 2:
        a,b = split(argv[0])
   -    print 'Usage: ' + b + ' <trackerurl> <file> [file...] [params...]'
   +    print 'Usage: ' + b + ' <file> [file...] [params...]'
        print
        print formatDefinitions(defaults, 80)
        print_announcelist_details()
   @@ -33,9 +33,9 @@
        exit(2)

    try:
   -    config, args = parseargs(argv[1:], defaults, 2, None)
   -    for file in args[1:]:
   -        make_meta_file(file, args[0], config, progress = prog)
   +    config, args = parseargs(argv[1:], defaults, 1, None)
   +    for file in args[0:]:
   +        make_meta_file(file, None, config, progress = prog)
    except ValueError, e:
        print 'error: ' + str(e)
        print 'run with no args for parameter explanations'

I then run the following script to verify the downloaded files, and generate the torrent files:


   $ cat cmp.sh 
   #!/bin/sh -
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/aarch64/Latest/
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
   # wget -c -r -l 1 -nd --limit-rate=800k https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/i386/Latest/
   # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,powerpc,powerpc-powerpc64,sparc64,arm64-aarch64}.asc
   # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-{amd64,i386,arm64-aarch64}-vm.asc
   # wget https://www.freebsd.org/releases/11.2R/CHECKSUM.SHA512-FreeBSD-11.2-RELEASE-arm-armv6-{BANANAPI,BEAGLEBONE,CUBIEBOARD,CUBIEBOARD2,CUBBOX-HUMMINGBOARD,GUMSTIX,PANDABOARD,RPI-B,RPI2,WANDBOARD}.asc

   grep -h '^SHA512' CHECK*.asc | sed -e 's/SHA512 (\(.*\)) = \(.*\)/\2 \1/' | sort -k 2 > sha512.from.asc

   while read hash fname; do
           if [ -e "$fname" ]; then
                   sigfile=`grep -l -- "$fname" *.asc | head -n 1`
                   echo checking "$fname", sig in: "$sigfile"
                   #res=`sha512 -q "$fname"`
                   res=`shasum -a 512 "$fname" | awk '{ print $1 }'`
                   echo "File is: $res"
                   if [ x"$res" != x"$hash" ]; then
                           echo missmatch!  "$fname"
                           exit 1
                   fi
                   if ! [ -e "$fname".torrent ]; then
                           btmaketrackerless.py "$fname"
                   fi
           else
                   echo missing "$fname"
                   exit 1
           fi
   done < sha512.from.asc

Once all the torrents have been generated, I then make the magnet links:


   $ cat btmakemagnet.sh 
   #!/bin/sh -

   # metainfo file.: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso.torrent
   # info hash.....: 06091dabce1296d11d1758ffd071e7109a92934f
   # file name.....: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
   # file size.....: 203161600 (775 * 262144 + 0)
   # announce url..: udp://tracker.openbittorrent.com:80
   # btshowmetainfo 20030621 - decode BitTorrent metainfo files

   for i in *.torrent; do
           btshowmetainfo.py "$i" | awk '
   $0 ~ "^info hash" { info = $3 }
   $0 ~ "^file name" { name = $3 }
   END {
           print "magnet:?xt=urn:btih:" info "&dn=" name
   }'
   done

I then create the magnet links file, and update the Torrents wiki page.

~~Sorry about the code formatting. I don’t know how to make it look better in blogger.~~This blog no longer uses blogger.

Unusable Insecurity

2018-03-16T00:00:00Z

Unusable Insecurity

Posted: March 16, 2018 at 1:40 PM

security

Many people claim that security is hard, and in many cases it is hard, but that isn’t an excuse to make it harder than it needs to be. There are many layers to security, but adding extra layers, or making security controls inscrutable is a great way to ensure insecurity. Security needs to be simple and straightforward to configure, and easy to understand. There may be knobs for advanced users, but defaults need to be simple and correct.

I recently looked at using S3 as a shared store for some data. I was using the account New Context created for me that had limited AWS permissions. Creating the S3 bucket was simple enough, and making it not-public was too, but then I wanted to create a user/API key that only had access to the S3 bucket. Per Amazon IAM Best Practices, you should not share your account, but create new users for access. It turns out that I did not have the CreateUser permission. I involved a co-worker who did have permissions to create the user. Adding another person to the task makes things more complex through communication and their availability to work on it instead of their normal work.

As part of creating a user, you have to figure out what the Policy that you need to assign to the user. Amazon provides some Bucket Policy Examples, but none of them is a simple policy on granting read and write permissions to the bucket. There is an Amazon Policy Generator for helping you to create the policies, but it doesn’t allow you to select buckets from your account (to simplify ARN [Amazon Resource Name] selection), and there are almost 70 actions provided in the selector. After some brief reading, I settled on a simple policy that I thought would allow the new user proper access: 4 permissions: PutObjects, GetObjects, ListObjects and RestoreObjects.

My co-worker created the user and applied the policy, but then I got an error handle code. Amazon does not provide an interface for turning on logging and/or querying why a request failed. Despite the error handle, I had ZERO insight into why the request failed. I could have involved AWS support, but now that would add yet another party in attempting to properly configure S3.

At this stage, I decided to give up, as I had already spent a few hours of my time, some of my co-worker’s time, and a couple weeks due to various delays due to availability and other work. In this case, storing the data in S3 was more of a nicety, and I decided that checking the data into a private git repo was adequate compared to the complexities involved in configuring S3. git was a tried and tested way to store data and restrict access while S3 for this usage was not, and hard to configure.

After I wrote this blog post, a coworker linked me to the blog post titled Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket. It is concerning that this blog post has not been integrated, nor linked to from any of the IAM or S3 documentation. This is a valuable resource that should not be hidden.

I’m clearly not the only one that has had issues configuring S3 buckets. The end of 2017 has shown a large number of organizations fail to properly secure their S3 buckets, leaving many terabytes of data open for public download. It is unacceptable that such a service is so difficult to configure. The site https://s3stupidity.com/ lists the large number of breaches, many of which are by large companies who should have the technical chops (and $$) to properly configure it.

Security controls need to be simple and clear. Their descriptions need to be accurate and concise in what they do, and how they do it. Amazon does have a number of good resources, but they do not have a comprehensive guide for what each permission does. You cannot blame users for security failures when you make it next to impossible to configure properly.

Edited to remove a couple extra words.

Adventures in Autobahn/WAMP Security

2017-09-17T00:00:00Z

Adventures in Autobahn/WAMP Security

Posted: September 17, 2017 at 12:21 PM

Or how security continues to suck because: It’s Hard and Someone Else’s Problem™¶

For a personal project, I’ve decided to use WAMP to move some events and messages around between different components. I decided on the AutoBahn libraries and Crossbar.io as the router. I was already somewhat familiar w/ AutoBahn from previous work, and the Crossbar.io router seems to just work. As a security person, I decided to evaluate how to make things as secure as possible.

First off, my projects must be both authenticated and encrypted. WAMP does not appear to have it’s own encryption layer, but it does have it’s own authentication layer. You really don’t want to have to trust two different authentication layersThe encryption layer must be authenticated, otherwise any attacker could MiTM the connection. Most uses of TLS make use of the CA system for authentication (which has serious issues in trust), and most web apps add their own authentication layer on top of it (not using Basic Auth, or other scheme). The issues w/ this is that if there is no binding between the two layers, the lower layer (application layer) cannot be sure that the upper layer has not been compromised., so being able to use TLS Channel Bindings would be an improvement. This would ensure that a strong authentication method in WAMP would ensure that the channel is properly encrypted. I received confirmation from the Crossbar.io team that it was present.

Autobahn and Crossbar.io supports a number of different authentication schemes. As I plan on putting this behind a reverse proxy (which I realize will have it’s own issues w/ channel binding), I wanted the strongest security binding between my client and the server (and I’m a glutton for punishment for using unproven tech). The only one that satisfies this requirement is WAMP-Cryptosign.

After I got basic functionality working to make sure things would be workable w/ this framework, I decided to start working on the authentication piece. First problem I ran into was that the AutoBahn|JS library does not support TLS channel binding. There is a good reason the library doesn’t support it, and it’s for a very bad reason. There is no support in the browser WebSocket API to query the channel binding information necessary. The fact that WebSockets was standardized after Channel bindings were demonstrates that the people involved in standardizing the web do not take security seriously. As usual, they assume that security is not their problem and leaves it up to someone else to solve (or at another layer).

Disappointed that I wouldn’t be able to use channel bindings w/ the web client for this project (I still had the crappy CA authentication of TLS, so not all was lost), I moved forward w/ CryptoSign. As has been demonstrated many times, the only way to get security baked in, is to make it as easy as possible to use. I’ve been long familiar w/ Crypto Box by djb (and used by the Autobahn libraries), and also the noise protocol (which my friend Trevor created). Both of these have goals of making it simple to let developers include security in their projects and not mess it up, resulting in a broken system. As currently implemented, Autobahn’s CryptoSign is most definitely not easy to use.

Though the documentation is decent, some examples are not present (client_ssh_key.py for example from WAMP-cryptosign Static Authentication). The ApplicationRunner helper class does not document how to make use of authentication. Though the static authentication page has examples, they make you write quite a bit of boiler plate.

Then even once you do that, you find out that the code doesn’t even work on Python 2.7 and have to fix it for them. Hopefully the pull request (PR) will not be ignored because of the failing CI tests, because the current CI tests are problems with their CI environment, and not the PR. For CI checks like this, it should only ding your PR on checks that are newly failing, and ignore any checks that were previously failing. This isn’t the first project that their CI environment was broken.

Even w/ the fixes in place, there is no documented method of extracting a public key from a generated ssh key. I will be adding a method to print this out.

If I (who knows cryptography decently) have to fix and spend hours making this work, it’s no wonder than everyone things that strong cryptography is hard. It is hard, but it shouldn’t be.

Installing and running NetBSD and OpenBSD under bhyve

2015-07-28T00:00:00Z

Installing and running NetBSD and OpenBSD under bhyve

Posted: July 28, 2015 at 8:37 PM

These instructions assume that you have downloaded the install ISO from the respective sources. These were doing with specific versions, and there may be minor changes with older and newer versions.

These instructions could possibly be more simple, such as not using separate device maps for grub-bhyve. These were testing on a month old HEAD.

There are other guides that cover most of this, and probably in more detail. The issue that I had was the exact commands to grub to load kernels was not well documented. Both of the images boot and are able to get DHCP leases and pass basic traffic.

Hope this helps others!

NetBSD¶

Install grub2-bhyve:
```
   pkg install grub2-bhyve
   
```

Create a file called instdev.map containing:


   (cd0) NetBSD-6.1.5-amd64.iso
   (hd1) netbsd.img

Create the file netbsd.img with the correct size:
```
   truncate -s 3g netbsd.img
   
```

Run the following commands (or put into a script file) under sh:


   MEM=512M
   VM=nbsd615
   bhyvectl --destroy --vm=$VM
   grub-bhyve -r cd0 -M $MEM -m instdev.map $VM <<EOF
   knetbsd -h -r cd0a
   (cd0)/netbsdboot
   EOF
   bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
       -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./netbsd.img \
       -s 4:0,ahci-cd,./NetBSD-6.1.5-amd64.iso \
       -l com1,stdio -c 2 -m $MEM $VM

This will run the installer, complete the installation.
Create a file called dev.map containing:
```
   (hd1) netbsd.img
   
```

Now in the future, to run NetBSD from the image, run the following commands:


   MEM=512M
   VM=nbsd615
   bhyvectl --destroy --vm=$VM
   grub-bhyve -r cd0 -M $MEM -m dev.map $VM <<EOF
   knetbsd -h -r ld0a
   (hd1,msdos1)/netbsdboot
   EOF
   bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
       -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./netbsd.img \
       -l com1,stdio -c 2 -m $MEM $VM

Profit!

OpenBSD¶

Install grub2-bhyve:
```
   pkg install grub2-bhyve
   
```

Create a file called instdev.map containing:


   (cd0) install57.iso
   (hd1) openbsd.img

Create the file openbsd.img with the correct size:
```
   truncate -s 3g openbsd.img
   
```

Run the following commands (or put into a script file) under sh:


   MEM=512M
   VM=obsd57
   bhyvectl --destroy --vm=$VM
   grub-bhyve -r cd0 -M $MEM -m instdev.map $VM <<EOF
   kopenbsd -h com0
   (cd0)/5.7/amd64/bsd.rdboot
   EOF
   bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
       -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./openbsd.img \
       -s 4:0,ahci-cd,./install57.iso \
       -l com1,stdio -c 2 -m $MEM $VM

This will run the installer, complete the installation.
Create a file called dev.map containing:
```
   (hd1) netbsd.img
   
```

Now in the future, to run OpenBSD from the image, run the following commands:


   MEM=512M
   VM=obsd57
   bhyvectl --destroy --vm=$VM
   grub-bhyve -r hd1 -M $MEM -m dev.map $VM <<EOF
   kopenbsd -h com0 -r sd0a
   (hd1,openbsd1)/bsdboot
   EOF
   bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
       -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./openbsd.img \
       -s 4:0,ahci-cd,./install57.iso \
       -l com1,stdio -c 2 -m $MEM $VM

Profit!

XML Schema Validation for the command line

2015-05-07T00:00:00Z

XML Schema Validation for the command line

Posted: May 7, 2015 at 2:17 PM

schema
xml

It turns out that unless you use a full fledge XML editor, validating your XML document against a schema is difficult. Most tools require you to specify a single schema file. If you have an XML document that contains more than one name space this doesn’t work too well as often, each name space is in a separate schema file.

The XML document has xmlns attributes which use a URI as the identifier. These URIs are for identifying it, and not a URL, so not able to be used. In fact, different cases in the URIs specify different name spaces even in the “host” part, though that is not the case with URLs. In order for validators to find the schema, the attribute xsi:schemaLocation is used to map the name space URIs to the URLs of the schema.

The xsi:schemaLocation mapping is very simple. It is simply a white space delimited list of URI/URL pairs. None of the command line tools that I used uses this attribute to make the schema validation simple. This includes xmllint Via WayBack Machine as original link it http only. which uses the libxml2 library. I also tried to use the Java XML library Xerces, but was unable to get it to work. Xerces did not provide a simple command line utility, and I couldn’t figure out the correct java command line to invoke the validator class.

My coworker, Patrick Via WayBack Machine as original link is now defunct., found the blog entry, Nokogiri XML schema validation with multiple schema files, which talks about using xs:import to have a single schema file support multiple name spaces. With this, we realized that we could finally get our XML document verified.

As I know shell scripting well, I decided to write a script to automate creating a unified schema and validate a document. The tools don’t cache the schema documents, requiring fetching the schema each time you want to validate the XML document. We did attempt to write the schema files to disk, and reuse those, but there are issues in that some schemas reference other resources in them. If the schema is not retrieved from the web, these internal resources are not retrieved also, causing errors when validating some XML documents.

With a little bit of help from xsltproc to extract xsi:schemaLocation, it wasn’t to hard to generate the schema document and provide it to xmllint.

The code (xmlval.sh):

#!/bin/sh -

cat <<EOF |
<?xml version="1.0"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>

 <xsl:output method="text"/>
 <xsl:template match="/">
  <xsl:value-of select="/*/@xsi:schemaLocation"/>
 </xsl:template>

</xsl:stylesheet>
EOF
    xsltproc - "$1" |
    sed -e 's/ */\
/g' |
    sed -e '/^$/d' |
    (echo '<?xml version="1.0" encoding="UTF-8"?>'
     echo '<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:nospace="nospace" targetNamespace="http://www.example.com/nospace">'
     while :; do
        if ! read a; then
            break
        fi
        if ! read b; then
            break
        fi
        echo '<xs:import namespace="'"$a"'" schemaLocation="'"$b"'"/>'
    done
    echo '</xs:schema>') |
    xmllint --noout --schema - "$1"

Though the script looks complicated, it is a straight forward pipeline:

Lines 3-16 provide the xslt document to xsltproc on line 17 to extract schema location attribute.
Lines 18-20 replace multiple spaces with new lines and deletes any blank lines. It should probably also handle tabs, but none of the documents that I have had tabs. After this, we now have the odd lines containing the URI of the name space, and the even lines contain the URL for the schema.
Lines 21 and 22 are the header for the new schema document.
Lines 23-31 pulls in these line pairs and create the necessary xs:import lines.
Line 32 provides the closing element for the schema document.
Line 33 gives the schema document to xmllint for validation.

Building bhyve Images using makefs and mkimg

2014-10-29T00:00:00Z

Building bhyve Images using makefs and mkimg

Posted: October 29, 2014 at 8:57 PM

Recently Neel Natu committed work to enable bhyve to run on AMD processors. My main development machine is an AMD A10-5700, so the commit enables me to use bhyve for testing.

EDIT: Anish Gupta did the work that Neel Natu committed. Thanks Anish!

I had previously built images using makefs and mkimg for a CF card for use in another machine, so being able to build images to use with bhyve makes sense.

First, you need to make sure that you have a complete source check out along with a completed buildworld and buildkernel. Then follow these steps:

Install world and distribution into a temporary directory using the NO_ROOT option:


    make installworld DESTDIR=<tmpdir> -DDB_FROM_SRC -DNO_ROOT
    make distribution DESTDIR=<tmpdir> -DDB_FROM_SRC -DNO_ROOT

This preps everything with the defaults as necessary.

Install a kernel either into a different directory (I do this) or into the same directory above:
```
    make installkernel DESTDIR=<tmpkerndir> -DNO_ROOT KERNCONF=<conf>
    
```
Make a directory with your custom configuration files. The basics are /etc/rc.conf and /etc/fstab and you might want /firstboot on there too. You will also need a METALOG file which contains the permissions for the files. This is just a standard mtree file, so you could use mtree to generate this instead of creating it by hand. The file contents are below.

Build the ufs image using the makeroot.sh script in the src tree at tools/tools/makeroot/makeroot.sh:


    /usr/src/tools/tools/makeroot/makeroot.sh  -e <custdir>/METALOG -e <tmpkerndir>/METALOG -p <tmpdir>/etc/master.passwd -s 2g ufs.img root

Build the disc image:


    mkimg -s gpt -b <tmpdir>/boot/pmbr -p freebsd-boot:=<tmpdir>/boot/gptboot -p freebsd-swap::1G -p freebsd-ufs:=ufs.img -o disc.img

Run the image:


    sh /usr/share/examples/bhyve/vmrun.sh -d disc.img vm0

There you have it. Besides running the image, all the other steps can be done as a normal user w/o root access.

EDIT: You also might want to include an /entropy file (populated with 4k from /dev/random) in your custom directory so that the image has a good seed for entropy at first boot for things such as sshd key generation.

File contents:

/etc/fstab:


  /dev/vtbd0p3    /               ufs     rw              1 1

Custom METALOG:


  #mtree 2.0
  ./etc/rc.conf type=file uname=root gname=wheel mode=0644
  ./etc/fstab type=file uname=root gname=wheel mode=0644
  ./firstboot type=file uname=root gname=wheel mode=0644

Python ctypes wrapper for FLAC

2014-03-15T00:00:00Z

Python ctypes wrapper for FLAC

Posted: March 15, 2014 at 5:11 PM

As many people know, I’ve a fan of Python and I have been using it for over 15 years now.

One of the recent improvements that Python has made is the inclusion of ctypes, which allows you to write a wrapper around shared libraries in Python making it much easier to integrate libraries. Previously you’d have to know the C Python API to get a module. There was SWIG, but if the library was complicated, it’d often not produce working code and even if it did, you’d have to hand tweak the output to get it to work the way you think it should.

One of the projects I’ve worked on is a UPnP media server. One of the features it has is the ability to decode a flac file and support seeking with in the file.

I have now released a package of the code: ctflac-1.0.tar.gz.

The one issue w/ ctypes is that some code can be very slow in Python. The FLAC library presents the sample data as arrays for each channel, though most libraries interleave the channel data. I have written a very small library (that is optional) interleave.c that does this interleaving in faster C code. In my tests, using the C code results in about a third of the CPU usage.

Hope this is useful for others!

CTF + ARMeb + debugging

2014-03-05T00:00:00Z

CTF + ARMeb + debugging

Posted: March 5, 2014 at 7:21 PM

I’ve been working on making the AVILA board work again with FreeBSD. Thanks to Jim from Netgate for sending me a board to do this work.

I still have a pending patch waiting to go through bde to fix an unaligned off_t store which gets things farther, but with the patch I’m getting a: panic: vm_page_alloc: page 0xc0805db0 is wired shortly after the machine launches the daemons.

I did work to get cross gdb working for armeb (committed in r261787 and r261788), but that didn’t help as there is no kernel gdb support on armeb. As I’m doing this debugging over the network, I can’t dump a core.

I didn’t feel like hand decoding a struct vm_page, so I thought of other methods, and one way is to use CTF to parse the data type and decode the data. I know python and ctypes, so I decided to wrap libctf and see what I could do.

Getting the initial python wrapper working was easy, but my initial test data was the kernel on my amd64 box that I am developing on. Now I needed to use real armeb CTF data. I point it to my kernel, and I get: “File uses more recent ELF version than libctf“. Ok, extract the CTF data from the kernel (ctf data is stored in a section named .SUNW_ctf) and work on that directly:

$ objcopy -O binary --set-section-flags optfiles=load,alloc -j .SUNW_ctf /tftpboot/kernel.avila.avila /dev/null
objcopy: /tftpboot/kernel.avila.avila: File format not recognized

Well, ok, that’s not too surprising since it’s an ARMEB binary, lets try:

$ /usr/obj/arm.armeb/usr/src.avila/tmp/usr/bin/objcopy -O binary --set-section-flags optfiles=load,alloc -j .SUNW_ctf /tftpboot/kernel.avila.avila /tmp/test.avila.ctf     
$ ls -l /tmp/test.avila.ctf 
-rwxr-xr-x  1 jmg  wheel  0 Mar  5 17:59 /tmp/test.avila.ctf

Hmm, that didn’t work too well, ok, lets just use dd to extract the data using info from objdump -x.

Ok, now that I’ve done that, I get:

ValueError: '/tmp/avila.ctf': File is not in CTF or ELF format

Hmm, why is that? Well, it turns out that the endian of the CTF data is wrong. The magic is cf f1, but the magic on amd64 is f1 cf, it’s endian swapped. That’s annoying. After spending some time trying to build an cross shared version of libctf, I find that it has the same issue.

After a bit of looking around, I discover the CTF can only ever read native endianness, but ctfmerge has a magic option that will write out endian swapped data if necessary depending upon the ELF file it’s putting in. This means that the CTF data in an armeb object file will be different depending upon the endian you compiled it on, so the object file isn’t cross compatible. But, this does mean that the data in the object files will be readable by libctf, just not the data written into the kernel.

So, I create a sacrificial amd64 binary:

$ echo 'int main() {}' | cc -o /tmp/avila2.ctf -x c -

And use ctfmerge to put the data in it:

$ ctfmerge -L fldkj -o /tmp/avila2.ctf /usr/obj/arm.armeb/usr/src.avila/sys/AVILA/*.o

and again use dd to extract the .SUNW_ctf section into a separate file.

With all this work, I finally have the CTF data in a format that libctf can parse, so, I try to parse some data. Now the interesting thing is that the CTF data does encode sizes of integers, but it uses the native arch’s pointer sizes for CTF_K_POINTER types, which means that pointers appear to be 8 bytes in size instead of the correct 4 bytes. A little more hacking on the ctf.py script to force all pointers to be 4 bytes, and a little help to convert ddb output to a string and finally, I have a dump of the struct vm_page that I was trying to get all along:

{'act_count': '\x00',
 'aflags': '\x00',
 'busy_lock': 1,
 'dirty': '\xff',
 'flags': 0,
 'hold_count': 0,
 'listq': {'tqe_next': 0xc0805e00, 'tqe_prev': 0xc06d18a0},
 'md': {'pv_kva': 3235856384,
        'pv_list': {'tqh_first': 0x0, 'tqh_last': 0xc0805de0},
        'pv_memattr': '\x00',
        'pvh_attrs': 0},
 'object': 0xc06d1878,
 'oflags': '\x04',
 'order': '\t',
 'phys_addr': 17776640,
 'pindex': 3572,
 'plinks': {'memguard': {'p': 0, 'v': 3228376932},
            'q': {'tqe_next': 0x0, 'tqe_prev': 0xc06d1f64},
            's': {'pv': 0xc06d1f64, 'ss': {'sle_next': 0x0}}},
 'pool': '\x00',
 'queue': '\xff',
 'segind': '\x01',
 'valid': '\xff',
 'wire_count': 1}

So, the above was produced w/ the final ctf.py script.