Sunday, May 23, 2010

OpenBSD QEMU setup scripts

Here is a pair of scripts I've used to setup harnesses for some testing of custom images I've made, these will require that you have sudo access to ifconfig, as well as have qemu installed. I also use non default tun interfaces, (not installed by default), to get these working go to /dev and run "./MAKEDEV tun10 tun11 tun12 tun13 tun14 tun15"

This first on is a harness setup script.

To run the setup script simply do "sh setup-vharness.sh"

This will configure 6 tun interfaces for the image to use, I use them as WAN/LAN/DMZ, but you can use however you want, change ip's in the script to suite your needs.

#!/bin/sh
# setup-vharness.sh
# script to setup qemu guest carp setup on an obsd box

SUDO=/usr/bin/sudo
MODE=ENABLE

usage() {
echo "usage: $0 [-d]" 1>&2
exit 2
}

start() {
echo MODE: $MODE
if [ $MODE = "DISABLE" ]; then
echo DeConfiguring CARP Harness
echo DeConfiguring tun10:
echo ${SUDO} ifconfig tun10 destroy
${SUDO} ifconfig tun10 destroy
echo DeConfiguring tun11:
echo ${SUDO} ifconfig tun11 destroy
${SUDO} ifconfig tun11 destroy
echo DeConfiguring tun12:
echo ${SUDO} ifconfig tun12 destroy
${SUDO} ifconfig tun12 destroy
echo DeConfiguring tun13:
echo ${SUDO} ifconfig tun13 destroy
${SUDO} ifconfig tun13 destroy
echo DeConfiguring tun14:
echo ${SUDO} ifconfig tun14 destroy
${SUDO} ifconfig tun14 destroy
echo DeConfiguring tun15:
echo ${SUDO} ifconfig tun15 destroy
${SUDO} ifconfig tun15 destroy
echo DeConfiguring bridge0:
echo ${SUDO} ifconfig bridge0 destroy
${SUDO} ifconfig bridge0 destroy
echo DeConfiguring bridge1:
echo ${SUDO} ifconfig bridge1 destroy
${SUDO} ifconfig bridge1 destroy
echo DeConfiguring bridge2:
echo ${SUDO} ifconfig bridge2 destroy
${SUDO} ifconfig bridge2 destroy
else
# make sure a tun interface is available
echo Configuring CARP Harness
echo Configuring tun10:
echo ${SUDO} ifconfig tun10 192.168.1.254 link0
${SUDO} ifconfig tun10 192.168.1.254 link0
echo Configuring tun11:
echo ${SUDO} ifconfig tun11 192.168.254.254 link0
${SUDO} ifconfig tun11 192.168.254.254 link0
echo Configuring tun12:
echo ${SUDO} ifconfig tun12 192.168.253.254 link0
${SUDO} ifconfig tun12 192.168.253.254 link0
echo Configuring tun13:
echo ${SUDO} ifconfig tun13 link0
${SUDO} ifconfig tun13 link0 up
echo Configuring bridge0:
echo ${SUDO} ifconfig bridge0 create
${SUDO} ifconfig bridge0 create
echo ${SUDO} brconfig bridge0 add tun10 add tun13 up
${SUDO} brconfig bridge0 add tun10 add tun13 up
echo Configuring tun14:
echo ${SUDO} ifconfig tun14 link0
${SUDO} ifconfig tun14 link0 up
echo Configuring bridge1:
echo ${SUDO} ifconfig bridge1 create
${SUDO} ifconfig bridge1 create
echo ${SUDO} brconfig bridge1 add tun11 add tun14 up
${SUDO} brconfig bridge1 add tun11 add tun14 up
echo Configuring tun15:
echo ${SUDO} ifconfig tun15 link0
${SUDO} ifconfig tun15 link0 up
echo Configuring bridge2:
echo ${SUDO} ifconfig bridge2 create
${SUDO} ifconfig bridge2 create
echo ${SUDO} brconfig bridge2 add tun12 add tun15 up
${SUDO} brconfig bridge2 add tun12 add tun15 up
fi
}

args=`getopt d $*`

set -- $args
while [ $# -gt 0 ]; do
case "$1" in
-d) MODE="DISABLE"
echo "Disable Mode"
;;
--) shift;
break
;;
esac
shift
done

start $1



This next script actually brings up the image file, it sets up the serial port to be redirected to a telnet session (all my firewall images use console port, you'll need to change this if you want to see the actual screen output: man qemu). This will also dump you into a telnet session connected to the serial of the device, be aware of that. To get this one to work, just do "sh setup-qemu-img.sh " The offset is to tell the script which CARP host your using, offset is where to start the nic configs at for this host.

So after running the above script to configure this, I just do the following:

sh setup-qemu.sh 0

Then I split my window in two using tmux "Ctrl-B S"

sh setup-qemu.sh 3

At this point I have both images running side by side, and can configure the individual guests to suit my needs.

!!!You may want to change some of the arguments you give to qemu to suit your needs, again see the qemu man page.


#!/bin/sh
# setup-qemu.sh
# script to start multiple qemus on a single box

SUDO=/usr/bin/sudo

# qemu args
IMAGE=$1
MEMORY=128
FLAGS=" -daemonize -no-kqemu -nographic -serial telnet:127.0.0.1:$((1010 + $2)),server,nowait -no-fd-bootchk"

NICFLAGS="-net nic,vlan=\$id,macaddr=\$mac -net tap,vlan=\$id,ifname=\$id,fd=\$fd"

getmac() {
mac="00:bd:`printf %02x $(($RANDOM % 256))`:"
mac="$mac`printf %02x $(($RANDOM % 256))`:"
mac="$mac`printf %02x $(($1 % 256))`:`printf %02x $(($2 % 255 + 4))`"
}

start() {
for id in 0 1 2; do
fd=$(($id + 3))
tun=tun$(($id + 10 + $2))
getmac 3 id
eval "nics=\"$nics $NICFLAGS\""
fds="$fds $fd<> /dev/$tun"

done

cmd="${SUDO} -C 5 qemu -m ${MEMORY} -hda ${IMAGE}${FLAGS}$nics$fds"
echo Running: ${SUDO} sh -c "$cmd"
${SUDO} sh -c "$cmd"
}

start $1 $2

telnet localhost $((1010 + $2))



To tear down the setup script simply do "sh setup-vharness.sh -d" after you've quit the qemu instance(s).

Tuesday, May 18, 2010

OpenBSD 4.7 Release pf.conf conversion tool

Today marks the release of OpenBSD 4.7, with this are many new changes outlined here. One of the changes requires pre 4.7 pf.conf files to be modified before they will load, changes to the nat/rdr/route-to/reply-to syntax are the reason for this. These changes are outlined here, normally these can be edited by hand fairly easily, but with many boxes to manage the following script seemed to save me a little time, this hasn't been tested extensively, but seems to work for me. If you find a bug let me know and I'll fix it.

Usage: prompt> perl pfconvert.pl /etc/pf.conf > /etc/pf.conf.new

!!! Note, make sure you verify the converted file will work, ie "pfctl -nf /etc/pf.conf.new"

#!/usr/bin/perl

# Read the config into a scaler
my $pfconf = do { local( @PFCONF, $/ ) = $ARGV[0] ; <> } ;

# Merge any multiline rules into a single line
$pfconf =~ s/\\\n/+multilinebreak+/g;

# Read the file into an array so we can process line by line
@PFCONF = split('\n', $pfconf);

foreach $line (@PFCONF) {
# Remove rdr-anchors for relayd ftp-proxy and tftp-proxy
if ($line =~ m/^nat-anchor|^rdr-anchor/) {
if ($line =~ m/relayd|ftp-proxy|tftp-proxy/) {
next;
}
}

# Convert rdr rules to new 4.7 syntax
if ($line =~ m/^rdr on/) {
$line =~ s/^rdr/match in/;
$line =~ s/->/rdr-to/;
}

# Convert nat rules to new 4.7 syntax
if ($line =~ m/^nat on/) {
$line =~ s/^nat/match out/;
$line =~ s/->/nat-to/;
}

# Convert binat rules to new 4.7 syntax
if ($line =~ m/^binat on/) {
$line =~ s/^binat/match/;
$line =~ s/->/binat-to/;
}

# Convert reply-to rules to new 4.7 syntax
if ($line =~ m/reply-to/) {
if ($line =~ m/\sto\s+(.*)/) {
$dest = $1;
}
if ($line =~ m/\s(\(.*\))/) {
$ifaddr = $1;
}
$line =~ s/reply-to .+/to $dest reply-to $ifaddr/;
}

# Convert route-to rules to new 4.7 syntax
if ($line =~ m/route-to/) {
if ($line =~ m/\sfrom\s+(.*)/) {
$dest = $1;
}
if ($line =~ m/\s(\(.*\))/) {
$ifaddr = $1;
}
$line =~ s/route-to .+/from $dest route-to $ifaddr/;
}

# Unmerge any multiline rules
$line =~ s/\+multilinebreak\+/\\\n/g;
print "$line\n";
}

Wednesday, May 12, 2010

OpenBSD Terminal Colors and tmux session on login

This is my .profile for my account on my OpenBSD boxes, this requires that you have the colorls and vim packages installed, but this will give you color coded directory listings, and syntax highlighted vim as well as dump you into a tmux session on login, this is very handy, allowing your session to get disconnected and not lose your work. Pretty simple really just replace your .profile with this in your home directory:

# $OpenBSD: dot.profile,v 1.4 2005/02/16 06:56:57 matthieu Exp $
#
# sh/ksh initialization

alias ls='colorls -G'
alias vi=vim

PATH=$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin:/usr/games:.
export PATH HOME TERM=xterm-256color

tmux attach || tmux new

Tuesday, May 11, 2010

Open Source WAN Emulation Howto

This is old, but I see people asking for it every now and then.

Quick Dummynet howto...

I needed to emulate a real wan connection for some testing I'm doing. This was a PITA to find out on my own, so I'm documenting everything.

This is just on a livecd right now, so you have to issue these commands every time you reboot, which sucks, but luckily it runs on BSD which, hardly ever needs to reboot.

First you'll want to grab a copy of the Frenzy Freebsd Live CD here:

http://frenzy.org.ua/en/index.shtml

After it boots, I issue these commands (note, i'm running this on a box with 3 nics, 1 for managment, 2 for the bridge) You could pull it off with just two nics as well, later I'm going to turn it into a router too, so it will do routing as well as introducing bw settings latency, loss, etc. Also on the Todo list is make a vmware image of all of this with scripts to easily change settings for the bridge/router...

For right now this is what I do to get it up and running:

#setup networking
ifconfig em0 172.16.62.11 up
route add default 172.16.62.254
ifconfig bridge create
ifconfig bridge0 addm em1 addm em2 up
ifconfig em1 up
ifconfig em2 up

#load dummynet
kldload dummynet

#ipfw commands
sysctl net.link.bridge.ipfw=1
ipfw add pipe 1 ip from any to any
ipfw pipe 1 config bw 2Mbits/s delay 30ms plr .01
ipfw list

# we need to make sure that the any to any rule with the pipe is the highest any to any rule..
ipfw delete 65000
ipfw list

# From Lucas: adds memory for network buffers
#the queue for dummynet can take a lot of memory and it behaves badly when it runs out

sysctl kern.ipc.nmbclusters=32768



#To watch traffic on the bridge, simply type
tcpdump -i bridge0

Nice, just found this to backup settings to a floppy:

http://frenzy.org.ua/en/releases/1.0/doc/doc_en-backup.html

Here are some links that I found helpful when figuring all this out:

http://info.iet.unipi.it/~luigi/ip_dummynet/
http://www.freebsd-howto.com/HOWTO/Ipfw-HOWTO
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html
http://www.freebsd.org/cgi/man.cgi?query=if_bridge&sektion=4

Google's Annotated Timeline Chart and Sql Server 2005

I had a small project where I needed to use the functionality of the Google Timeline Annotated Chart from the Google API. The provided charts in the API are pretty nifty, but most of the data that I would want to display is kept in a DB somewhere, I needed a way to have simple SQL queries build a chart for me on the fly, below is what I came up with.

First attempt was to use php to query the DB format all my data in a way that Google wanted and then spit out the html into a div, this worked but was an extra step that was unnecessary.

For this example we'll go ahead and say I have a table of cities and their high temperatures for a given day in the last week, this will be a very simple table with 3 columns consisting of Timestamp, City, Temp. (In reality my table structure was a little more complicated, but for this example, a simple table called TempData should suffice).

The following stored procedure will get all the values needed from our database and return a varchar that consists of the html needed to display the graph, we can then put that value into a div and the graph will be displayed.




USE [DATABASENAME]
GO

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[sp_google_temperature_trends]

AS

DECLARE
@Timestamp varchar(50),
@Year varchar(50),
@Month varchar(50),
@Day varchar(50),
@Hour varchar(50),
@Min varchar(50),
@City varchar(50),
@Temp varchar(50),
@avgTemp varchar(50),
@t int,
@html varchar(max),
@hideCityHtml varchar(max)

/* First We'll print out the top of the container info for the goolge annotated timeline graph, for google api info see:
http://code.google.com/apis/visualization/documentation/gallery/annotatedtimeline.html
*/

SET @html = '








' + @hideCityHtml +'
'