Friday, March 16, 2018

Unusable Insecurity

Many people claim that security is hard, and in many cases it is hard, but that isn't an excuse to make it harder than it needs to be. There are many layers to security, but adding extra layers, or making security controls inscrutable is a great way to ensure insecurity. Security needs to be simple and straightforward to configure, and easy to understand. There may be knobs for advanced users, but defaults need to be simple and correct.

I recently looked at using S3 as a shared store for some data. I was using the account New Context created for me that had limited AWS permissions. Creating the S3 bucket was simple enough, and making it not-public was too, but then I wanted to create a user/API key that only had access to the S3 bucket. Per Amazon IAM Best Practices, you should not share your account, but create new users for access. It turns out that I did not have the CreateUser permission. I involved a co-worker who did have permissions to create the user. Adding another person to the task makes things more complex through communication and their availability to work on it instead of their normal work.

As part of creating a user, you have to figure out what the Policy that you need to assign to the user. Amazon provides some Bucket Policy Examples, but none of them is a simple policy on granting read and write permissions to the bucket. There is an Amazon Policy Generator for helping you to create the policies, but it doesn't allow you to select buckets from your account (to simplify ARN [Amazon Resource Name] selection), and there are almost 70 actions provided in the selector. After some brief reading, I settled on a simple policy that I thought would allow the new user proper access: 4 permissions: PutObjects, GetObjects, ListObjects and RestoreObjects.

My co-worker created the user and applied the policy, but then I got an error handle code. Amazon does not provide an interface for turning on logging and/or querying why a request failed. Despite the error handle, I had ZERO insight into why the request failed. I could have involved AWS support, but now that would add yet another party in attempting to properly configure S3.

At this stage, I decided to give up, as I had already spent a few hours of my time, some of my co-worker's time, and a couple weeks due to various delays due to availability and other work. In this case, storing the data in S3 was more of a nicety, and I decided that checking the data into a private git repo was adequate compared to the complexities involved in configuring S3. git was a tried and tested way to store data and restrict access while S3 for this usage was not, and hard to configure.

After I wrote this blog post, a coworker linked me to the blog post titled Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket. It is concerning that this blog post has not been integrated, nor linked to from any of the IAM or S3 documentation. This is a valuable resource that should not be hidden.

I'm clearly not the only one that has had issues configuring S3 buckets. The end of 2017 has shown a large number of organizations fail to properly secure their S3 buckets, leaving many terabytes of data open for public download. It is unacceptable that such a service is so difficult to configure. The site lists the large number of breaches, many of which are by large companies who should have the technical chops (and $$) to properly configure it.

Security controls need to be simple and clear. Their descriptions need to be accurate and concise in what they do, and how they do it. Amazon does have a number of good resources, but they do not have a comprehensive guide for what each permission does. You cannot blame users for security failures when you make it next to impossible to configure properly.

Edited to remove a couple extra words.

Sunday, September 17, 2017

Adventures in Autobahn/WAMP Security

Or how security continues to suck because: It's Hard and Someone Else's Problem™

For a personal project, I've decided to use WAMP to move some events and messages around between different components.  I decided on the AutoBahn libraries and as the router.  I was already somewhat familiar w/ AutoBahn from previous work, and the router seems to just work.  As a security person, I decided to evaluate how to make things as secure as possible.

First off, my projects must be both authenticated and encrypted.  WAMP does not appear to have it's own encryption layer, but it does have it's own authentication layer.  You really don't want to have to trust two different authentication layers1, so being able to use TLS Channel Bindings would be an improvement.  This would ensure that a strong authentication method in WAMP would ensure that the channel is properly encrypted.  I received confirmation from the team that it was present.

Autobahn and supports a number of different authentication schemes.  As I plan on putting this behind a reverse proxy (which I realize will have it's own issues w/ channel binding), I wanted the strongest security binding between my client and the server (and I'm a glutton for punishment for using unproven tech).  The only one that satisfies this requirement is WAMP-Cryptosign.

After I got basic functionality working to make sure things would be workable w/ this framework, I decided to start working on the authentication piece.  First problem I ran into was that the AutoBahn|JS library does not support TLS channel binding.  There is a good read the library doesn't support it, and it's for a very bad reason.  There is no support in the browser WebSocket API to query the channel binding information necessary.  The fact that WebSockets was standardized after Channel bindings were demonstrates that the people involved in standardizing the web do not take security seriously.  As usual, they assume that security is not their problem and leaves it up to someone else to solve (or at another layer).

Disappointed that I wouldn't be able to use channel bindings w/ the web client for this project (I still had the crappy CA authentication of TLS, so not all was lost), I moved forward w/ CryptoSign.  As has been demonstrated many times, the only way to get security baked in, is to make it as easy as possible to use.  I've been long familiar w/ Crypto Box by djb (and used by the Autobahn libraries), and also the noise protocol (which my friend Trevor created).  Both of these have goals of making it simple to let developers include security in their projects and not mess it up, resulting in a broken system.  As currently implemented, Autobahn's CryptoSign is most definitely not easy to use.  

Though the documentation is decent, some examples are not present ( for example from WAMP-cryptosign Static Authentication).  The ApplicationRunner helper class does not document how to make use of authentication.  Though the static authentication page has examples, they make you write quite a bit of boiler plate.

Then even once you do that, you find out that the code doesn't even work on Python 2.7 and have to fix it for them.  Hopefully the pull request (PR) will not be ignored because of the failing CI tests, because the current CI tests are problems with their CI environment, and not the PR.  For CI checks like this, it should only ding your PR on checks that are newly failing, and ignore any checks that were previously failing.  This isn't the first project that their CI environment was broken.

Even w/ the fixes in place, there is no documented method of extracting a public key from a generated ssh key.  I will be adding a method to print this out.

If I (who knows cryptography decently) have to fix and spend hours making this work, it's no wonder than everyone things that strong cryptography is hard.  It is hard, but it shouldn't be.

1 The encryption layer must be authenticated, otherwise any attacker could MiTM the connection.  Most uses of TLS make use of the CA system for authentication (which has serious issues in trust), and most web apps add their own authentication layer on top of it (not using Basic Auth, or other scheme).  The issues w/ this is that if there is no binding between the two layers, the lower layer (application layer) cannot be sure that the upper layer has not been compromised.

Tuesday, July 28, 2015

Installing and running NetBSD and OpenBSD under bhyve

These instructions assume that you have downloaded the install ISO from the respective sources.  These were doing with specific versions, and there may be minor changes with older and newer versions.

These instructions could possibly be more simple, such as not using separate device maps for grub-bhyve.   These were testing on a month old HEAD.

There are other guides that cover most of this, and probably in more detail.  The issue that I had was the exact commands to grub to load kernels was not well documented.  Both of the images boot and are able to get DHCP leases and pass basic traffic.

Hope this helps others!


  1. Install grub2-bhyve:
    pkg install grub2-bhyve
  2. Create a file called containing:(cd0) NetBSD-6.1.5-amd64.iso
    (hd1) netbsd.img
  3. Create the file netbsd.img with the correct size:
    truncate -s 3g netbsd.img
  4. Run the following commands (or put into a script file) under sh:MEM=512MVM=nbsd615
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r cd0 -M $MEM -m $VM <<EOFknetbsd -h -r cd0a (cd0)/netbsdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
        -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./netbsd.img \
        -s 4:0,ahci-cd,./NetBSD-6.1.5-amd64.iso \
        -l com1,stdio -c 2 -m $MEM $VM
  5. This will run the installer, complete the installation.
  6. Create a file called containing:
    (hd1) netbsd.img
  7. Now in the future, to run NetBSD from the image, run the following commands:
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r cd0 -M $MEM -m $VM <<EOFknetbsd -h -r ld0a (hd1,msdos1)/netbsdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
        -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./netbsd.img \
        -l com1,stdio -c 2 -m $MEM $VM
  8. Profit!


  1. Install grub2-bhyve:
    pkg install grub2-bhyve
  2. Create a file called containing:
    (cd0) install57.iso(hd1) openbsd.img
  3.  Create the file openbsd.img with the correct size:
    truncate -s 3g openbsd.img
  4. Run the following commands (or put into a script file) under sh:MEM=512MVM=obsd57
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r cd0 -M $MEM -m $VM <<EOFkopenbsd -h com0 (cd0)/5.7/amd64/bsd.rdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \    -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./openbsd.img \    -s 4:0,ahci-cd,./install57.iso \    -l com1,stdio -c 2 -m $MEM $VM
  5. This will run the installer, complete the installation.
  6. Create a file called containing:
    (hd1) netbsd.img
  7. Now in the future, to run OpenBSD from the image, run the following commands:
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r hd1 -M $MEM -m $VM <<EOFkopenbsd -h com0 -r sd0a (hd1,openbsd1)/bsdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
        -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./openbsd.img \
        -s 4:0,ahci-cd,./install57.iso \
        -l com1,stdio -c 2 -m $MEM $VM
  8. Profit!

Thursday, May 7, 2015

XML Schema Validation for the command line

It turns out that unless you use a full fledge XML editor, validating your XML document against a schema is difficult.  Most tools require you to specify a single schema file.  If you have an XML document that contains more than one name space this doesn't work too well as often, each name space is in a separate schema file.

The XML document has xmlns attributes which use a URI as the identifier.  These URIs are for identifing it, and not a URL, so not able to be used.  In fact, different cases in the URIs specify different name spaces even in the "host" part, though that is not the case with URLs.  In order for validators to find the schema, the attribute xsi:schemaLocation is used to map the name space URIs to the URLs of the schema.

The xsi:schemaLocation mapping is very simple.  It is simply a white space delimited list of URI/URL pairs.  None of the command line tools that I used uses this attribute to make the schema validation simple.  This includes xmllint which uses the libxml2 library.  I also tried to use the Java XML library Xerces, but was unable to get it to work.  Xerces did not provide a simple command line utility, and I couldn't figure out the correct java command line to invoke the validator class.

My coworker, Patrick, found the blog entry, Nokogiri XML schema validation with multiple schema files, which talks about using xs:import to have a single schema file support multiple name spaces.  With this, we realized that we could finally get our XML document verified.

As I know shell scripting well, I decided to write a script to automate creating a unified schema and validate a document.  The tools don't cache the schema documents, requiring fetching the schema each time you want to validate the XML document.  We did attempt to write the schema files to disk, and reuse those, but there are issues in that some schemas reference other resources in them.  If the schema is not retrieved from the web, these internal resources are not retrieved also, causing errors when validating some XML documents.

With a little bit of help from xsltproc to extract xsi:schemaLocation, it wasn't to hard to generate the schema document and provide it to xmllint.

The code (

1:  #!/bin/sh -  
3:  cat <<EOF |  
4:  <?xml version="1.0"?>  
5:  <xsl:stylesheet version="1.0"  
6:      xmlns:xsl=""  
7:      xmlns:xsi=""  
8:  >  
10:   <xsl:output method="text"/>  
11:   <xsl:template match="/">  
12:    <xsl:value-of select="/*/@xsi:schemaLocation"/>  
13:   </xsl:template>  
15:  </xsl:stylesheet>  
16:  EOF  
17:      xsltproc - "$1" |  
18:      sed -e 's/ */\  
19:  /g' |  
20:      sed -e '/^$/d' |  
21:      (echo '<?xml version="1.0" encoding="UTF-8"?>'  
22:       echo '<xs:schema xmlns:xs="" xmlns:nospace="nospace" targetNamespace="">'  
23:       while :; do  
24:          if ! read a; then  
25:              break  
26:          fi  
27:          if ! read b; then  
28:              break  
29:          fi  
30:          echo '<xs:import namespace="'"$a"'" schemaLocation="'"$b"'"/>'  
31:      done  
32:      echo '</xs:schema>') |  
33:      xmllint --noout --schema - "$1"  

Though the script looks complicated, it is a straight forward pipeline:

  1. Lines 3-16 provide the xslt document to xsltproc on line 17 to extract schema location attribute.
  2. Lines 18-20 replace multiple spaces with new lines and deletes any blank lines.  It should probably also handle tabs, but none of the documents that I have had tabs.  After this, we now have the odd lines containing the URI of the name space, and the even lines contain the URL for the schema.
  3. Lines 21 and 22 are the header for the new schema document.
  4. Lines 23-31 pulls in these line pairs and create the necessary xs:import lines.
  5. Line 32 provides the closing element for the schema document
  6. Line 33 gives the schema document to xmllint for validation.

Wednesday, October 29, 2014

Building bhyve Images using makefs and mkimg

Recently Neel Natu committed work to enable bhyve to run on AMD processors.  My main development machine is an AMD A10-5700, so the commit enables me to use bhyve for testing.

EDIT: Anish Gupta did the work that Neel Natu commited.  Thanks Anish!

I had previously built images using makefs and mkimg for a CF card for use in another machine, so being able to build images to use with bhyve makes sense.

First, you need to make sure that you have a complete source check out along with a completed buildworld and buildkernel.  Then follow these steps:
  1. Install world and distribution into a temporary directory using the NO_ROOT option:
    make installworld DESTDIR=<tmpdir> -DDB_FROM_SRC -DNO_ROOT
    make distribution DESTDIR=<tmpdir> -DDB_FROM_SRC -DNO_ROOT
    This preps everything with the defaults as necessary.
  2. Install a kernel either into a different directory (I do this) or into the same directory above:
    make installkernel DESTDIR=<tmpkerndir> -DNO_ROOT KERNCONF=<conf>
  3. Make a directory with your custom configuration files.  The basics are /etc/rc.conf and /etc/fstab and you might want /firstboot on there too.  You will also need a METALOG file which contains the permissions for the files.  This is just a standard mtree file, so you could use mtree to generate this instead of creating it by hand.  The file contents are below.
  4. Build the ufs image using the script in the src tree at tools/tools/makeroot/
    /usr/src/tools/tools/makeroot/  -e <custdir>/METALOG -e <tmpkerndir>/METALOG -p <tmpdir>/etc/master.passwd -s 2g ufs.img root
  5. Build the disc image:
    mkimg -s gpt -b <tmpdir>/boot/pmbr -p freebsd-boot:=<tmpdir>/boot/gptboot -p freebsd-swap::1G -p freebsd-ufs:=ufs.img -o disc.img
  6. Run the image:
    sh /usr/share/examples/bhyve/ -d disc.img vm0
There you have it.   Besides running the image, all the other steps can be done as a normal user w/o root access.

EDIT: You also might want to include an /entropy file (populated with 4k from /dev/random) in your custom directory so that the image has a good seed for entropy at first boot for things such as sshd key generation.

File contents:
  • /etc/fstab:
    /dev/vtbd0p3    /               ufs     rw              1 1
  • Custom METALOG:#mtree 2.0
    ./etc/rc.conf type=file uname=root gname=wheel mode=0644
    ./etc/fstab type=file uname=root gname=wheel mode=0644
    ./firstboot type=file uname=root gname=wheel mode=0644

Saturday, March 15, 2014

Python ctypes wrapper for FLAC

As many people know, I've a fan of Python and I have been using it for over 15 years now.

One of the recent improvements that Python has made is the inclusion of ctypes, which allows you to write a wrapper around shared libraries in Python making it much easier to integrate libraries.  Previously you'd have to know the C Python API to get a module.  There was SWIG, but if the library was complicated, it'd often not produce working code and even if it did, you'd have to hand tweak the output to get it to work the way you think it should.

One of the projects I've worked on is a UPnP media server.  One of the features it has is the ability to decode a flac file and support seeking with in the file.

I have now released a package of the code:

The one issue w/ ctypes is that some code can be very slow in Python.  The FLAC library presents the sample data as arrays for each channel, though most libraries interleave the channel data.  I have written a very small library (that is optional) interleave.c that does this interleaving in faster C code.  In my tests, using the C code results in about a third of the CPU usage.

Hope this is useful for others!

Wednesday, March 5, 2014

CTF + ARMeb + debugging

I've been working on making the AVILA board work again with FreeBSD.  Thanks to Jim from Netgate for sending me a board to do this work.

I still have a pending patch waiting to go through bde to fix an unaligned off_t store which gets things farther, but with the patch I'm getting a: panic: vm_page_alloc: page 0xc0805db0 is wired shortly after the machine launches the daemons.

I did work to get cross gdb working for armeb (committed in r261787 and r261788), but that didn't help as there is no kernel gdb support on armeb.  As I'm doing this debugging over the network, I can't dump a core.

I didn't feel like hand decoding a struct vm_page, so I thought of other methods, and one way is to use CTF to parse the data type and decode the data.  I know python and ctypes, so I decided to wrap libctf and see what I could do.

Getting the initial python wrapper working was easy, but my initial test data was the kernel on my amd64 box that I am developing on.  Now I needed to use real armeb CTF data.  I point it to my kernel, and I get: "File uses more recent ELF version than libctf".  Ok, extract the CTF data from the kernel (ctf data is stored in a section named .SUNW_ctf) and work on that directly:
$ objcopy -O binary --set-section-flags optfiles=load,alloc -j .SUNW_ctf /tftpboot/kernel.avila.avila /dev/null
objcopy: /tftpboot/kernel.avila.avila: File format not recognized

Well, ok, that's not too surprising since it's an ARMEB binary, lets try:
$ /usr/obj/arm.armeb/usr/src.avila/tmp/usr/bin/objcopy -O binary --set-section-flags optfiles=load,alloc -j .SUNW_ctf /tftpboot/kernel.avila.avila /tmp/test.avila.ctf     
$ ls -l /tmp/test.avila.ctf 
-rwxr-xr-x  1 jmg  wheel  0 Mar  5 17:59 /tmp/test.avila.ctf

Hmm, that didn't work too well, ok, lets just use dd to extract the data using info from objdump -x.

Ok, now that I've done that, I get:
ValueError: '/tmp/avila.ctf': File is not in CTF or ELF format

Hmm, why is that?  Well, it turns out that the endian of the CTF data is wrong.  The magic is cf f1, but the magic on amd64 is f1 cf, it's endian swapped.  That's annoying.  After spending some time trying to build an cross shared version of libctf, I find that it has the same issue.

After a bit of looking around, I discover the CTF can only ever read native endianness, but ctfmerge has a magic option that will write out endian swapped data if necessary depending upon the ELF file it's putting in.  This means that the CTF data in an armeb object file will be different depending upon the endian you compiled it on, so the object file isn't cross compatible.  But, this does mean that the data in the object files will be readable by libctf, just not the data written into the kernel.

So, I create a sacrificial amd64 binary:
$ echo 'int main() {}' | cc -o /tmp/avila2.ctf -x c -

And use ctfmerge to put the data in it:
$ ctfmerge -L fldkj -o /tmp/avila2.ctf /usr/obj/arm.armeb/usr/src.avila/sys/AVILA/*.o

and again use dd to extract the .SUNW_ctf section into a separate file.

With all this work, I finally have the CTF data in a format that libctf can parse, so, I try to parse some data.  Now the interesting thing is that the CTF data does encode sizes of integers, but it uses the native arch's pointer sizes for CTF_K_POINTER types, which means that pointers appear to be 8 bytes in size instead of the correct 4 bytes.  A little more hacking on the script to force all pointers to be 4 bytes, and a little help to convert ddb output to a string and finally, I have a dump of the struct vm_page that I was trying to get all along:
{'act_count': '\x00',
 'aflags': '\x00',
 'busy_lock': 1,
 'dirty': '\xff',
 'flags': 0,
 'hold_count': 0,
 'listq': {'tqe_next': 0xc0805e00, 'tqe_prev': 0xc06d18a0},
 'md': {'pv_kva': 3235856384,
        'pv_list': {'tqh_first': 0x0, 'tqh_last': 0xc0805de0},
        'pv_memattr': '\x00',
        'pvh_attrs': 0},
 'object': 0xc06d1878,
 'oflags': '\x04',
 'order': '\t',
 'phys_addr': 17776640,
 'pindex': 3572,
 'plinks': {'memguard': {'p': 0, 'v': 3228376932},
            'q': {'tqe_next': 0x0, 'tqe_prev': 0xc06d1f64},
            's': {'pv': 0xc06d1f64, 'ss': {'sle_next': 0x0}}},
 'pool': '\x00',
 'queue': '\xff',
 'segind': '\x01',
 'valid': '\xff',
 'wire_count': 1}

So, the above was produced w/ the final script.