Tuesday, July 28, 2015

Installing and running NetBSD and OpenBSD under bhyve

These instructions assume that you have downloaded the install ISO from the respective sources.  These were doing with specific versions, and there may be minor changes with older and newer versions.

These instructions could possibly be more simple, such as not using separate device maps for grub-bhyve.   These were testing on a month old HEAD.

There are other guides that cover most of this, and probably in more detail.  The issue that I had was the exact commands to grub to load kernels was not well documented.  Both of the images boot and are able to get DHCP leases and pass basic traffic.

Hope this helps others!

NetBSD

  1. Install grub2-bhyve:
    pkg install grub2-bhyve
  2. Create a file called instdev.map containing:(cd0) NetBSD-6.1.5-amd64.iso
    (hd1) netbsd.img
  3. Create the file netbsd.img with the correct size:
    truncate -s 3g netbsd.img
  4. Run the following commands (or put into a script file) under sh:MEM=512MVM=nbsd615
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r cd0 -M $MEM -m instdev.map $VM <<EOFknetbsd -h -r cd0a (cd0)/netbsdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
        -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./netbsd.img \
        -s 4:0,ahci-cd,./NetBSD-6.1.5-amd64.iso \
        -l com1,stdio -c 2 -m $MEM $VM
  5. This will run the installer, complete the installation.
  6. Create a file called dev.map containing:
    (hd1) netbsd.img
  7. Now in the future, to run NetBSD from the image, run the following commands:
    MEM=512MVM=nbsd615
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r cd0 -M $MEM -m dev.map $VM <<EOFknetbsd -h -r ld0a (hd1,msdos1)/netbsdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
        -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./netbsd.img \
        -l com1,stdio -c 2 -m $MEM $VM
  8. Profit!

OpenBSD

  1. Install grub2-bhyve:
    pkg install grub2-bhyve
  2. Create a file called instdev.map containing:
    (cd0) install57.iso(hd1) openbsd.img
  3.  Create the file openbsd.img with the correct size:
    truncate -s 3g openbsd.img
  4. Run the following commands (or put into a script file) under sh:MEM=512MVM=obsd57
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r cd0 -M $MEM -m instdev.map $VM <<EOFkopenbsd -h com0 (cd0)/5.7/amd64/bsd.rdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \    -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./openbsd.img \    -s 4:0,ahci-cd,./install57.iso \    -l com1,stdio -c 2 -m $MEM $VM
  5. This will run the installer, complete the installation.
  6. Create a file called dev.map containing:
    (hd1) netbsd.img
  7. Now in the future, to run OpenBSD from the image, run the following commands:
    MEM=512MVM=obsd57
    bhyvectl --destroy --vm=$VM
    grub-bhyve -r hd1 -M $MEM -m dev.map $VM <<EOFkopenbsd -h com0 -r sd0a (hd1,openbsd1)/bsdbootEOFbhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc \
        -s 2:0,virtio-net,tap3 -s 3:0,virtio-blk,./openbsd.img \
        -s 4:0,ahci-cd,./install57.iso \
        -l com1,stdio -c 2 -m $MEM $VM
  8. Profit!

Thursday, May 7, 2015

XML Schema Validation for the command line

It turns out that unless you use a full fledge XML editor, validating your XML document against a schema is difficult.  Most tools require you to specify a single schema file.  If you have an XML document that contains more than one name space this doesn't work too well as often, each name space is in a separate schema file.

The XML document has xmlns attributes which use a URI as the identifier.  These URIs are for identifing it, and not a URL, so not able to be used.  In fact, different cases in the URIs specify different name spaces even in the "host" part, though that is not the case with URLs.  In order for validators to find the schema, the attribute xsi:schemaLocation is used to map the name space URIs to the URLs of the schema.

The xsi:schemaLocation mapping is very simple.  It is simply a white space delimited list of URI/URL pairs.  None of the command line tools that I used uses this attribute to make the schema validation simple.  This includes xmllint which uses the libxml2 library.  I also tried to use the Java XML library Xerces, but was unable to get it to work.  Xerces did not provide a simple command line utility, and I couldn't figure out the correct java command line to invoke the validator class.

My coworker, Patrick, found the blog entry, Nokogiri XML schema validation with multiple schema files, which talks about using xs:import to have a single schema file support multiple name spaces.  With this, we realized that we could finally get our XML document verified.

As I know shell scripting well, I decided to write a script to automate creating a unified schema and validate a document.  The tools don't cache the schema documents, requiring fetching the schema each time you want to validate the XML document.  We did attempt to write the schema files to disk, and reuse those, but there are issues in that some schemas reference other resources in them.  If the schema is not retrieved from the web, these internal resources are not retrieved also, causing errors when validating some XML documents.

With a little bit of help from xsltproc to extract xsi:schemaLocation, it wasn't to hard to generate the schema document and provide it to xmllint.

The code (xmlval.sh):

1:  #!/bin/sh -  
2:    
3:  cat <<EOF |  
4:  <?xml version="1.0"?>  
5:  <xsl:stylesheet version="1.0"  
6:      xmlns:xsl="http://www.w3.org/1999/XSL/Transform"  
7:      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
8:  >  
9:    
10:   <xsl:output method="text"/>  
11:   <xsl:template match="/">  
12:    <xsl:value-of select="/*/@xsi:schemaLocation"/>  
13:   </xsl:template>  
14:    
15:  </xsl:stylesheet>  
16:  EOF  
17:      xsltproc - "$1" |  
18:      sed -e 's/ */\  
19:  /g' |  
20:      sed -e '/^$/d' |  
21:      (echo '<?xml version="1.0" encoding="UTF-8"?>'  
22:       echo '<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:nospace="nospace" targetNamespace="http://www.example.com/nospace">'  
23:       while :; do  
24:          if ! read a; then  
25:              break  
26:          fi  
27:          if ! read b; then  
28:              break  
29:          fi  
30:          echo '<xs:import namespace="'"$a"'" schemaLocation="'"$b"'"/>'  
31:      done  
32:      echo '</xs:schema>') |  
33:      xmllint --noout --schema - "$1"  

Though the script looks complicated, it is a straight forward pipeline:

  1. Lines 3-16 provide the xslt document to xsltproc on line 17 to extract schema location attribute.
  2. Lines 18-20 replace multiple spaces with new lines and deletes any blank lines.  It should probably also handle tabs, but none of the documents that I have had tabs.  After this, we now have the odd lines containing the URI of the name space, and the even lines contain the URL for the schema.
  3. Lines 21 and 22 are the header for the new schema document.
  4. Lines 23-31 pulls in these line pairs and create the necessary xs:import lines.
  5. Line 32 provides the closing element for the schema document
  6. Line 33 gives the schema document to xmllint for validation.