It takes me about three times the amount of time to do a transcript as the podcast’s length (which is about 80 minutes), so generally a couple of nights’ work for a single SGU podcast. I guess I might be able to manage about two podcasts a week, and given that there are currently a bit over 350 podcasts in the archive, and a new one released each week, it’ll probably take me about seven years to finish them all! So… I created a wiki, sgutranscripts.org. Now we’ll just have to see if the community gets behind it, otherwise I could be here for a long time.

The conventional wisdom when running a headless server with VirtualBox is that you need to install this proprietary extension pack from Oracle. This is fine until you want to use the server in production: as the PUEL only covers you for personal use and evaluation, you must purchase licenses. You can either pay £34 per user or £670 per “socket” (which has quite a convoluted definition). This gets you USB2 and RDP support.

However, there is another way, at least when it comes to RDP support. Chapter 7 of the VirtualBox manual covers running virtual machines on a remote host, either with VBoxManage command (which offeres similar functionality to the VirtualBox GUI) or the VBoxHeadless command, which appears to be the backend binary that VBoxManage calls out to to do the actual work. Now, chapter 7 covers in detail how to connect to remote VMs using the proprietary extensions, but there’s one feature that they conveniently failed to mention: the built-in VNC server.

man VBoxHeadless
VBOXHEADLESS(1)            User Commands             VBOXHEADLESS(1)

NAME
       VBoxHeadless - x86 virtualization solution

DESCRIPTION
    Oracle  VM  VirtualBox Headless Interface (C) 2008-2011 Oracle
    Corporation All rights reserved.

Usage:
    -s, -startvm, --startvm <name|uuid>
           Start given VM (required argument)
    -n, --vnc
           Enable the built in VNC server
    -m, --vncport <port>
           TCP port number to use for the VNC server
    -o, --vncpass <pw>
           Set the VNC server password
    -v, -vrde, --vrde on|off|config
           Enable (default) or disable the VRDE  server  or  don't
           change the setting
    -e,  -vrdeproperty,  --vrdeproperty  <name=[value]>
    Set a VRDE property:
           "TCP/Ports" - comma-separated list of  ports  the  VRDE
           server can bind to. Use a dash between two port numbers
           to specify a range "TCP/Address"  -  interface  IP  the
           VRDE server will bind to
    -c, -capture, --capture
           Record the VM screen output to a file
    -w, --width
           Frame width when recording
    -h, --height
           Frame height when recording
    -r, --bitrate
           Recording bit rate when recording
    -f, --filename
           File  name when recording.  The codec used will be cho‐
           sen based on the file extension

VBoxHeadless                January 2011             VBOXHEADLESS(1)

Right! So we can start a virtual machine and forward its root console (or main video or out-of-band console or whatever you want to call it) over the network with a VNC server. In my example I’ll use screen to keep my VMs running when I log out. I’ll start a couple of VMs as an example:

screen
VBoxHeadless --startvm 'Ubuntu' --vnc --vncport 5900
# hit "ctrl-a c" to open a new terminal within screen
VBoxHeadless --startvm 'XP1' --vnc --vncport 5901 --vncpass vnc2xrdp

Great! We don’t need the proprietary expansion pack to do this. However, there are a few drawbacks to this approach:

• VNC sucks over slow network connections, it’d be much nicer to use the more modern RDP protocol
• You have to keep the command running so you have to use screen or nohup or equivalent, which is a bit less convenient than using VBoxManage
• If the VNC server crashes (which I’ve had happen only once so far in testing, when changing screen resolution in a Windows XP guest), the whole VM goes down with it
• You need to make sure you only bind one VNC server to each port. As far as I can tell if you try to bind a second one to a port that’s already in use, the VM still starts up but you have no way of interacting with it!

We can actually work around the first limitation, by using the xrdp program to “translate” the VNC protocol into RDP. I’m on Ubuntu, so I have the luxury of installing xrdp the easy way. On the VirtualBox server machine:

sudo apt-get install xrdp

Now we configure it to use the existing VNC servers that we previously spawned with VBoxHeadless. One neat thing here is that xrdp uses a single RDP port to manage multiple VNC connections:

/etc/xrdp/xrdp.ini

[globals]
bitmap_cache=yes
bitmap_compression=yes
port=3389
crypt_level=low
channel_code=1

[xrdp1]
name=VBox-Ubuntu
lib=libvnc.so
ip=127.0.0.1
port=5900

[xrdp2]
name=VBox-XP1
lib=libvnc.so
username=
password=vnc2xrdp
ip=127.0.0.1
port=5901

So we have two VirtalBox VMs running, Ubuntu and XP1. In the example above, I started the ubuntu VNC without a password, so I’ve left out the username/password entries. The XP1 connection is protected with the password vnc2xrdp. You can also use the value ask and xrdp will prompt for a username/password for connecting to VNC. Note that VNC passwords are generally insecure, so it’d probably be best to protect the vnc ports using firewalling. It doesn’t appear to be possible to bind the VNC server to only the loopback device (at least from the man page above).

So now all that’s left to do is to connect to the RDP port using one of the myriad RDP clients for linux (I’m using Remmina, but there are heaps of options). You can then choose the VNC connection you want xrdp to connect to and you’re away!

xrdp login screen

Of course, SSH local port forwarding is your friend if you’re doing any of this through firewalls or over insecure connections. Remmina actually includes this functionality, or you can forward the remote port to your local machine with something like:

ssh -L 3389:localhost:3389 vboxservermachine

So there you are, RDP connections to remote VirtualBox VMs without the proprietary Oracle extensions; all free software!

Have fun.

sudo apt-get install mysql-workbench

Read on if you still want to compile from source.

Right now (2012-04-04), Ubuntu 12.04 hasn’t been released yet, and so there is no binary package from Oracle of MySQL Workbench for Precise. I managed to get the MySQL Workbench binaries for Oneiric to run, by manually installing libzip1_0.9.3-1_amd64.deb from Oneiric, but this wasn’t stable (crashed as soon as I tried to run a SQL Query).

So I decided to build from source. Here’s how I did it:

wget http://www.mirrorservice.org/sites/ftp.mysql.com/Downloads/MySQLGUITools/mysql-workbench-gpl-5.2.38-src.tar.gz
md5sum mysql-workbench-gpl-5.2.38-src.tar.gz
# should be cd2a0cec9dffd5465b6999f5d9c8de78 (from http://dev.mysql.com/downloads/workbench/#downloads).
tar xvzf mysql-workbench-gpl-5.2.38-src.tar.gz
cd mysql-workbench-gpl-5.2.38-src
# from http://bugs.mysql.com/bug.php?id=63898
fgrep -rlZ pkglib_DATA --include Makefile.am . | xargs -0 sed -i 's/pkglib_DATA/pkgdata_DATA/g'
# from https://bugzilla.redhat.com/show_bug.cgi?id=750023
vim ./modules/db.mysql.sqlparser/src/mysql_sql_parser_fe.cpp

change line 23 from:

#include <glib/gunicode.h>

to

#include <glib.h>

apt-get update
# from http://ubuntuforums.org/showthread.php?t=1792874 plus updates from comments below.
sudo apt-get install build-essential autoconf automake libtool libzip-dev libxml2-dev libsigc++-2.0-dev libglade2-dev libgtkmm-2.4-dev libgl1-mesa-dev libmysqlclient-dev uuid-dev liblua5.1-dev libpcre3-dev g++ libgnome2-dev libgtk2.0-dev libpango1.0-dev libcairo2-dev libsqlite3-dev python-dev libboost-dev libctemplate-dev
./autogen.sh
# I use -j4 below to use all four of my CPUs, set this appropriately for your setup.
make -j 4 install DESTDIR=/home/`echo $USER`/mysql-workbench
~/mysql-workbench/usr/local/bin/mysql-workbench

The informatics team here at MalariaGEN have been working with ec2 since before I joined them. So naturally, it’s one technology with which I’ve had to come to grips in the course of doing my job. For me, EC2 had a fairly steep learning curve, and after spending a while trying to learn it through doing, I decided that I would just have to spend some time getting properly to grips with how things worked. As part of that I decided to document it in a way that I’d not yet seen on the web: logically, comprehensively, explaining all the strange concepts and quirks that were clouding my understanding and stopping me from getting my job done efficiently.

We’re an Ubuntu shop here, or at least we’re moving to becoming one (the work I’m doing to move us off our ageing RHEL4 servers will be a topic for a future post), so this story will be from an Ubuntu point of view.

So first of all, we need to join up to the Amazon EC2 service. You can do this using your existing Amazon account, or create a completely new Amazon account to attach your EC2 subscription to.

So, we go to https://aws.amazon.com/ and sign up, putting in our credit card details and so forth. Eventually, we will end up at the AWS Management Console which is where we will first set up our credentials. But therein lay my first problem: I didn’t understand the various different types of credentials.

EC2 Credentials

In EC2, there are multiple different kinds of credential, Amazon uses slightly non-standard nomenclature, and it’s not always clear which credential is required for a given application.

1. Signon credentials: These are the email address/password pair that you use when you sign up. You use these to sign on to the AWS Management Console, and can be considered the “master” credentials as they allow you to regenerate all other types of credentials.

2. Access Credentials: There are three types: access keys, X.509 certificates and key pairs. The first and second type allow you to connect to the Amazon APIs. Which type of credential depends on which API and tool you are using. Some APIs and tools support both options, whereas others support just one. The third type is SSH public/private key pairs that are used for initial logins to newly created instances.

1. access keys: Symmetric key encryption. These are for making requests to AWS product REST or Query APIs. Can be obtained/regenerated from the Access Keys tab on the AWS Security Credentials page.
2. X.509 certificates: Public key encryption. Use X.509 certificates to make secure SOAP protocol requests to AWS service APIs. These are the credentials you will use when using the command-line ec2 api tools. Can be obtained/regenerated from the X.509 Certificates tab on the AWS Security Credentials page.
3. key pairs: SSH key pairs. When you create an instance, Amazon inserts the public key of your SSH key pair into your new instance so that you can log in using your private key. You can add new SSH key pairs through the AWS Management Console by clicking on Key Pairs under Networking and Security in the Navigation pane and then the Create Key Pair button. After specifying a name you will be prompted to download and save your private key. EC2 stores the public portion of your key pair, and inserts it into /home/ubuntu/.ssh/authorized_keys when you create your instance. If you lose this private key, it cannot be downloaded again; you will need to regenerate a new key pair.

Now that we understand credentials, we can get them all set up. First, lets set up our SSH key pair:
Go to the AWS Management Console, Networking and Security in the Navigation pane and then click the Create Key Pairbutton (and save your SSH private key in e.g. ~/.ec2/ec2.pem). The private key lives on your computer, while the public key lives in your AWS account.

You will also need to set up your Amazon API credentials. Go to Account->Security Credentials

1. click X.509 Certificates tab
2. Create a new certificate
3. Download the private key and the certificate (save them in e.g. ~/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem and ~/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem).
4. Make your credential files private: chmod go-rwx ~/.ec2/*.pem
5. Scroll to the bottom of the page and note your account ID (a number of the form XXXX-XXXX-XXXX).

Setting up the command line tools

OK, we now have our X.509 certificate and private key for accessing the ec2-api calls (we’re going to use the linux command line tools, but you can use the AWS console to perform a subset of these tasks, and there are GUI applications that can be used too).

Make sure you have multiverse enabled and run the following command:

sudo apt-get install ec2-api-tools

Next, we set up some variables in our shell environment so that we don’t always have to specify them as options to the command line tools:

export EC2_URL=https://ec2.eu-west-1.amazonaws.com
export EC2_KEYPAIR_EU_WEST_1=rwh-ec2 # ec2 SSH private key (name only, not the file)
export EC2_PRIVATE_KEY=$HOME/.ec2/rwh-ec2/pk-UZGAF4E4GLDA7DIDGUVWFIAKJTPK7MTX.pem # ec2 X.509 private key
export EC2_CERT=$HOME/.ec2/rwh-ec2/cert-UZGAF4E4GLDA7DIDGUVWFIAKJTPK7MTX.pem # ec2 X.509 certificate
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/

test that the environment is set up correctly:

ec2-describe-images -o self -o amazon

EC2 security groups

Security groups allow you to specify firewalling rules for your instances. These firewalling rules are independent of, and in addition to, the software firewalling provided by the instance’s operating system (iptables in the case of modern Ubuntu systems). Security groups must be defined before you create the instances that you would like to be members of those security groups. You specify the security groups to add an instance to at creation time with the -g option to the ec2-run-instances command. You cannot add an existing instance to a security group.

How you set up your security groups is up to you. You may choose to set up security groups that correspond to server functions, or have a separate security group for each instance. An instance may be a member of multiple security groups. If you don’t specify any security groups when you instantiate an instance, it will be added to the default security group. Our examples use the default security group, but keep in mind that this means that this causes an inability to set up firewalling rules in a granular fashion.

If you wish to create a more complex security group configuration, you can do so with these commands:

ec2-add-group  -d
ec2-delete-group
ec2-describe-group [ ...]

See the Using Security Groups section of the User Guide for Amazon EC2for more information.

Instantiating an image

The first thing to do is work out what kind of instance that you want to instantiate. See Amazon EC2 Instance Types or Instance Families and Types for descriptions of the available instance types, and Amazon EC2 Pricing for the current pricing of instances, data transfer and storage. Next, decide what release of Ubuntu you’d like, then look up the AMI code from the links in this table of official Ubuntu EC2 images:

I’ve chosen a t1.micro and the latest and greatest 11.04 for my example. So let’s do it! Let’s finally create an actual, running instance:

ec2-run-instances ami-4290a636 --instance-type t1.micro --region eu-west-1 --key ${EC2_KEYPAIR_EU_WEST_1}

RESERVATION r-e61ead90 153873855856 default
INSTANCE i-3c75064a ami-4290a636 pending rwh-ec2 0 t1.micro 2011-06-16T14:59:24+0000 eu-west-1c aki-4feec43b monitoring-disabled ebs paravirtual

Now we can find out a little bit about our new virtual machine:

ec2-describe-instances

RESERVATION r-e61ead90 153873855856 default
INSTANCE i-3c75064a ami-4290a636 ec2-46-137-45-101.eu-west-1.compute.amazonaws.com ip-10-227-167-14.eu-west-1.compute.internal running rwh-ec2 0 t1.micro 2011-06-16T14:59:24+0000 eu-west-1c aki-4feec43b monitoring-disabled 46.137.45.101 10.227.167.14 ebs paravirtual
BLOCKDEVICE /dev/sda1 vol-1be88672 2011-06-16T14:59:44.000Z

Now we need to open up the SSH port so that we can connect to the machine:

ec2-authorize default -p 22

GROUP default
PERMISSION default ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0

Now we should be able to see it in the aws web console, and SSH to it:

ssh -i ~/.ec2/rwh-ec2/rwh-ec2.pem ubuntu@ec2-56-237-35-121.eu-west-1.compute.amazonaws.com

The authenticity of host 'ec2-56-237-35-121.eu-west-1.compute.amazonaws.com (46.137.45.101)' can't be established.
RSA key fingerprint is 09:99:28:1c:5f:1a:89:f3:f9:6b:b4:83:88:b4:4c:23.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-56-237-35-121.eu-west-1.compute.amazonaws.com,56.237.35.121' (RSA) to the list of known hosts.

And we’re in! Now we can begin to set up and use the instance just like any other Ubuntu server machine.

You will be billed as long the host is running, so you will probably want to shut it down when you’re done. Note that each partial instance-hour consumed will be billed as a full hour.

ec2-terminate-instances <instance_id>

Fry the onions, carrots and celery until the onions are translucent. Add the curry powder and fry a bit more. Add the stock, lentils, tomatoes and garlic. Bring to a boil then reduce heat and simmer for 25 mins or until lentils are cooked. Salt and pepper to taste AND CHILLI. Eat with buttered toast.

Having said that, there’s one thing that I really don’t like: the way that it’s virtually impossible to tell what you have selected in windows other than the one that happens to be focused.  This is because the focused elements get completely desaturated, like this:

Which items are selected? It's anyone's guess!

For comparison, here’s the same window when it has focus:

We can see what's selected... so long as the window has focus.

This is a particular problem when you have two windows containing elements that you’d like to compare.  If you select these elements, and try to compare them by eye, you’ll often not be able to tell which one you selected in the unfocused window.

So I set out to change this by fiddling with the arcane theme files.  The one you need to change is:

/usr/share/themes/Ambiance/gtk-2.0/gtkrc

So, the way I did it was to add a new colour for my elements.  Line 1:

gtk-color-scheme = "base_color:#ffffff\nfg_color:#4c4c4c\ntooltip_fg_color:#ffffff\nselected_bg_color:#f07746\nselected_fg_color:#FFFFFF\ntext_color:#3C3C3C\nbg_color:#F2F1F0\ntooltip_bg_color:#000000\nlink_color:#DD4814"

becomes:

gtk-color-scheme = "base_color:#ffffff\nfg_color:#4c4c4c\ntooltip_fg_color:#ffffff\nselected_bg_color:#f07746\nselected_fg_color:#FFFFFF\ntext_color:#3C3C3C\nbg_color:#F2F1F0\ntooltip_bg_color:#000000\nlink_color:#DD4814\ntest-color:#f5d8b1"

Note the addition of the “\ntest-color:#f5d8b1″ at the end of the line. Next, we change the elements we want to that colour.  A bit of trial and error here and I found which one.  Line 93 was:



base[ACTIVE]      = shade (0.94, @bg_color)

and becomes:

base[ACTIVE]      = @test-color

Voila!  Our unfocused window now looks much better:

Now we can tell exactly what's selected, and it even fits in nicely with the theme.

Now just open up the Appearance application, switch to a different theme, then back to Ambiance to load the changes, and enjoy!

1. download and install Ubuntu 11.04 on a USB key
- I used the Startup Disk Creator on another Ubuntu machine to create the bootable USB key.

2. Put in the bootable Ubuntu USB key and boot the ideapad.  Press Fn-F11 to get the boot menu, and choose the USB boot device.  The screen will be garbled, seems like an issue with the kernel framebuffer driver.  Just hit enter, and the Ubuntu setup process will start fine.

3. Connect to an ethernet cable as wireless doesn’t work (says that it’s disabled by hardware switch despite not being, needs more investigation).

Update: Turns out that wireless works in 10.10… OK I have found a workaround for the wireless.  It even works with suspend/resume. Update 2: there’s an even simpler method in the comments, though I’ve not tested it myself.   Here’s how you can connect using wireless:
- connect via the wired ethernet port
- apt-get install wicd
- apt-get remove network-manager
- disconnect the wired ethernet port
- rfkill unblock all
- start wicd from the applications menu, then use it to connect to your wireless connection.

You can have the “rfkill unblock all” command run automatically on startup  by adding it to /etc/rc.local above the “exit 0″ line.  That’s all you should need to do to have the wireless connection come up automatically when you log in.

Seems like this might be the combination of a couple of bugs, one a configuration problem that necessitates the use of rfkill unblock all, and the other a bug in network-manager.  I will file some bug reports when I get time and post an update here.

4. After the Ubuntu 11.04 environment starts up, double click on the desktop icon to start the install process.  You’ll eventually read the message: “This computer currently has Windows 7 on it.  What would you like to do?”
- I first did the “Install Ubuntu alongside Windows 7″ option and shrunk the windows partition.  As I later learned, this doesn’t work because GRUB 2 fails, we need to set up the partitions in a GRUB legacy friendly setup (it requires a boot partition, formatted ext2, as it doesn’t understand ext4 yet; I’m not sure what the status of its ext3 support is).  So I restarted the install process and:
- chose the “Something else” option.  I deleted the ext4 and swap partitions from the previous install and then used the resulting empty space to set it up like this:
- /dev/sda6 ext2 /boot 255MB
- /dev/sda7 ext4 /    13684MB
- /dev/sda8 swap
- device boot loader installation: /dev/sda6
- proceed through the install, setting up your location and user details, etc.

5. Once the install completes, you will be prompted to restart, however we need to set up GRUB legacy first, so:
- start a terminal (click the button in the top left, type terminal and hit enter, or try ctrl-alt-t).  Then use the following commands (these assume my partition setup from above, modify them if yours differs):
- sudo su
- mkdir /mnt/sda7
- mount /dev/sda7 /mnt/sda7
- cd /mnt/sda7
- mount /dev/sda6 ./boot
- mount –bind /sys ./sys
- mount –bind /proc ./proc
- mount –bind /dev ./dev
- chroot .
In plain English, what we just did was mount the new partitions that we created, in the same configuration that they will be in the new system.  Then we bound all the special filesystems into this, and then we used “chroot” a magical command that moves our terminal into our newly setup system, almost as if we’d just booted into it.  So all our following commands will affect the new system instead of our temporary live USB boot.

6. Follow these instructions to download to grub legacy: https://help.ubuntu.com/community/Grub2#Uninstalling%20GRUB%202
- apt-get update ; apt-get install grub
- update-grub
- yes to generate list
- grub-install /dev/sda6
We’ve now installed grub legacy in /dev/sda6, which is our ubuntu /boot partition.  We now need to get the Windows 7 bootloader to chain-boot this.  We do this using a little utility called EasyBCD, though it’s possible to modify the Windows config files manually if you prefer.

7. Reboot into windows 7.
- download, install and run Easy BCD
- Add new entry
- Linux/BSD tab
- Type: GRUB (Legacy)
- Name: Ubuntu 11.04
- Partition 5 (Linux -244 MiB)
- Click Add Entry
- Edit Boot Menu
- Boot default OS after: change to 4 seconds.
- Save Settings.

8. Reboot, and you should see a boot menu with Windows 7 and Ubuntu 11.04.  Done!

Prerequisites for gridengine: Most of the problems that I saw with the Debianised gridengine system are due to a lack of these prerequisites:

1. check the hosts file for localhost.localdomain type entries. If these are present, they will cause host communication to fail. Ensure that, at minimum, there is an entry in the hosts file of the master for each exec node, and in the hosts file of the exec nodes there should be an entry for the master. For example:

I will set up a cluster between my desktop machine, KWIAT22 and my laptop, caleb.
/etc/hosts on KWIAT22 contains:

127.0.0.1       localhost
#127.0.0.1      localhost.localdomain   localhost
129.67.46.129   KWIAT22
129.67.46.255   caleb

plus some other irrelevant entries. Note that localhost.localdomain is commented out.
/etc/hosts on caleb contains:

127.0.0.1       caleb
#127.0.0.1      localhost.localdomain   localhost
129.67.46.255   caleb
129.67.46.129   KWIAT22

Note again, the localhost.localdomain entry has been commented out.

2. Java is required for inter-host communication. We will use Sun Java, as it is assumed to be most compatible with Sun GridEngine. Edit /etc/apt/sources.list and uncomment the entries for the partner repository:

deb http://archive.canonical.com/ubuntu maverick partner
deb-src http://archive.canonical.com/ubuntu maverick partner

Then install the JRE:

apt-get install sun-java6-jre

Check which version of java we’ve got selected:

root@caleb:~# java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.1) (6b22-1.10.1-0ubuntu1)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

From that we can see that I still have OpenJDK selected, so we change that:

root@caleb:~# update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                      Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-6-openjdk/jre/bin/java   1061      auto mode
  1            /usr/lib/jvm/java-6-openjdk/jre/bin/java   1061      manual mode
  2            /usr/lib/jvm/java-6-sun/jre/bin/java       63        manual mode

Press enter to keep the current choice[*], or type selection number: 2
update-alternatives: using /usr/lib/jvm/java-6-sun/jre/bin/java to provide /usr/bin/java (java) in manual mode.
root@caleb:~# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

Now that we have these prerequisites satisfied, we can install the relevant gridengine packages. Installing gridengine on Ubuntu systems is made simple by the packages. We can install the packages on the master node (in our case KWIAT22):

apt-get install gridengine-client gridengine-qmon gridengine-exec gridengine-master

Configure SGE automatically? Yes
SGE cell name: default
SGE master hostname: KWIAT22 (this should be the fully qualified domain name of the SGE master, not localhost)

Output will typically look something like this:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  gridengine-common
The following NEW packages will be installed:
  gridengine-client gridengine-common gridengine-exec gridengine-master gridengine-qmon
0 upgraded, 5 newly installed, 0 to remove and 37 not upgraded.
Need to get 0 B/18.7 MB of archives.
After this operation, 44.8 MB of additional disk space will be used.
Do you want to continue [Y/n]?
Preconfiguring packages ...
Selecting previously deselected package gridengine-common.
(Reading database ... 372804 files and directories currently installed.)
Unpacking gridengine-common (from .../gridengine-common_6.2u5-1ubuntu1_all.deb) ...
Selecting previously deselected package gridengine-client.
Unpacking gridengine-client (from .../gridengine-client_6.2u5-1ubuntu1_amd64.deb) ...
Selecting previously deselected package gridengine-exec.
Unpacking gridengine-exec (from .../gridengine-exec_6.2u5-1ubuntu1_amd64.deb) ...
Selecting previously deselected package gridengine-master.
Unpacking gridengine-master (from .../gridengine-master_6.2u5-1ubuntu1_amd64.deb) ...
Selecting previously deselected package gridengine-qmon.
Unpacking gridengine-qmon (from .../gridengine-qmon_6.2u5-1ubuntu1_amd64.deb) ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Setting up gridengine-common (6.2u5-1ubuntu1) ...

Creating config file /etc/default/gridengine with new version
Setting up gridengine-client (6.2u5-1ubuntu1) ...
Setting up gridengine-exec (6.2u5-1ubuntu1) ...
error: communication error for "KWIAT22/execd/1" running on port 6445: "can't bind socket"
error: commlib error: can't bind socket (no additional information available)
..........................
critical error: abort qmaster registration due to communication errors
daemonize error: child exited before sending daemonize state
Setting up gridengine-master (6.2u5-1ubuntu1) ...
Initializing cluster with the following parameters:
 => SGE_ROOT: /var/lib/gridengine
 => SGE_CELL: default
 => Spool directory: /var/spool/gridengine/spooldb
 => Initial manager user: sgeadmin
Initializing spool (/var/spool/gridengine/spooldb)
Initializing global configuration based on /usr/share/gridengine/default-configuration
Initializing complexes based on /usr/share/gridengine/centry
Initializing usersets based on /usr/share/gridengine/usersets
Adding user sgeadmin as a manager
Cluster creation complete
Setting up gridengine-qmon (6.2u5-1ubuntu1) ...

Note that the execd cannot bind the socket. This occurs because of a left-over execd that failed to stop from a previous install. It also results if you don’t have java installed, as the execd won’t respond to /etc/init.d/gridengine-exec stop without java. Also, if you’re doing an apt-get purge gridengine-* to get back to a fresh slate, typically the execd will not be stopped properly, despite being removed from the system. This can be fixed by:

root@KWIAT22:~# ps aux |grep sge
sgeadmin 22244 0.0 0.0 135172 4940 ? Sl 17:42 0:00 /usr/lib/gridengine/sge_qmaster
sgeadmin 24272 0.0 0.0  58688 2500 ? Sl May16 0:22 /usr/lib/gridengine/sge_execd
root@KWIAT22:~# kill 24272
root@KWIAT22:~# /etc/init.d/gridengine-exec start
root@KWIAT22:~# /etc/init.d/gridengine-master restart
 * Restarting Sun Grid Engine Master Scheduler sge_qmaster

The logfiles we can use for tracking down problems in communication between the qmaster and execd processes are not in the standard debian/ubuntu locations. Instead, they are stored in /var/spool/gridengine/execd/messages for the qmaster and /tmp/execd_messages.[pid] or /var/spool/gridengine/execd/messages for the execd processes. The log messages for our previous socket problem look like this (/tmp/execd_messages.24107):

05/16/2011 20:17:16|  main|KWIAT22|E|communication error for "KWIAT22/execd/1" running on port 6445: "can't bind socket"
05/16/2011 20:17:17|  main|KWIAT22|E|commlib error: can't bind socket (no additional information available)
05/16/2011 20:17:45|  main|KWIAT22|C|abort qmaster registration due to communication errors
05/16/2011 20:17:47|  main|KWIAT22|W|daemonize error: child exited before sending daemonize state

If you see any lines containing |E| then you have an error that must be addressed. Any lines with |W| are warnings, and it’s probably wise to fix those too.

On the exec nodes:

apt-get install gridengine-exec

Configure SGE automatically? yes
SGE cell name: default
SGE master hostname: KWIAT22

After installing, you will see the following error in the /tmp/exed_messages.[pid] file and the process will exit:

05/18/2011 17:53:00|  main|caleb|E|getting configuration: denied: host "caleb" is neither submit nor admin host
05/18/2011 17:53:05|  main|caleb|C|can't get configuration qmaster - terminating

This occurs because the master doesn’t yet know about the exec node. We need to set up a basic configuration on the master. We will use the documentation in /usr/share/doc/gridengine-common/README.Debian, which I will duplicate here, to form the basis of our configuration:

Once you've installed SGE, you'll need to do at least some minimal
cluster configuration.

Quickstart
==========

 * Install gridengine-master, gridengine-exec and gridengine-client
   on the appropriate hosts.

 * Initially, only the sgeadmin user has admin privileges

 * It is suggested that you add yourself as a manager and
   perform the rest of these tasks as your own user:
   + sudo -u sgeadmin qconf -am myuser

 * and to a userlist:
   + qconf -au myuser users

 * Add a submission host:
   + qconf -as myhost.mydomain

 * Add an execution host:
   + qconf -ae
   You will now be prompted for information about the execution host.

 * Add a new host group:
   + qconf -ahgrp @allhosts

 * Add the exec host to the @allhosts list:
   + qconf -aattr hostgroup hostlist myhost.mydomain @allhosts

 * Add a queue:
   + qconf -aq main.q

 * Add the host group to the queue:
   + qconf -aattr queue hostlist @allhosts main.q

 * Make sure there is a slot allocated to the execd:
   + qconf -aattr queue slots "[myhost.mydomain=1]" main.q

 * Running qstat -f should then show you the execd waiting for jobs

The commands that I ran in my example:

sudo su
sudo -u sgeadmin qconf -am rwh
exit
qconf -au rwh users
qconf -as KWIAT22
qconf -ahgrp @allhosts  # just save the file without modifying it
qconf -aattr hostgroup hostlist KWIAT22 @allhosts
qconf -aq main.q # just save the file without modifying it
qconf -aattr queue hostlist @allhosts main.q
qconf -aattr queue slots "4, [KWIAT22=3]" main.q # 4 by default for all nodes, 3 specifically for KWIAT22, which leaves 1 of the 4 cpus free for the master process

we then add caleb as a submit and exec host:

qconf -as caleb
qconf -ae # change the hostname entry to caleb
qconf -aattr hostgroup hostlist KWIAT22 @allhosts

Once this is done, we need to start the execd on caleb

/etc/init.d/gridengine-exec start

Check that it doesn’t create a log file in /tmp/execd_messages.[pid]. If it doesn’t then it’s happy! Back on our master node, a qstat -f should now show us all set up. You can use the GUI qmon tool to get a better look at the setup. To use qmon, you must ssh to the master node with X11 forwarding enabled:

ssh -X hostname
qmon

Click the queue control button and then the Hosts tab. If the exec nodes are communicating properly with the master, you should see them listed there, and they should NOT have dashes for the information columns. If a node does show dashes, it’s not communicating correctly, and you’ll need to go look in the log files for the reason. Note that if java is not installed, the communication between nodes will not work, and this may or may not show up in the log files.

Failure to communicate with the exec node, caleb

Success communicating with the exec node, caleb, after installing and selecting Sun Java.

Next, we need to set up a parallel environment. This will allow gridengine to start processes on the remote exec nodes. We can do this with qmon, though it’s also possible with the CLI tools. In qmon, click the bottom left button, Parallel Environment Configuration. Click Add. In our example, we’re setting up the simplest form of parallel environment, which doesn’t include any message passing functionality. Set the Name to simple_pe. In our case, we have two 4-core machines, with one core reserved for the master process, so we have 7 slots. The rest we leave as default values, just click OK, then click done. Now click the top, second from the left, button Cluster Queues. On the Cluster Queues tab, click main.q, then click Modify. Click the parallel environment tab, then click simple_pe and move it over to the referenced PEs box. Click OK, and Done.

Lastly, you need to set up passwordless ssh access from the master node to the exec nodes for the users of the gridengine system. This is left as an exercise for the reader, but you might start with learning about OpenSSH key management.

[ $ii -le 21 ] is shorthand for “test $ii -le 21″

# using a for loop
for ii in {1..21}
do
        wget "http://helms-deep.cable.nu/~rwh/blog/wp-content/uploads/2011/03/img${ii}.jpg"
done
 
# using a while loop
ii=1
while [ $ii -le 21 ] # spaces are required
do
        wget "http://helms-deep.cable.nu/~rwh/blog/wp-content/uploads/2011/03/img${ii}.jpg"
        ii=`expr $ii + 1` # spaces are required
done

For an interesting discussion: why did I use ii for my incrementer instead of i?

2. Note the double quotes, which are required to escape the filenames which contain special characters (spaces and parentheses). It is always good practice to wrap variables that contain filenames in double quotes for this reason.

for file in img*.jpg
do
        mv "$file" "`basename "$file" .jpg` (modified).jpg"
done

3. First, make sure imagemagick is installed:

sudo apt-get install imagemagick

for file in img*\ \(modified\).jpg 
do 
        convert -resize 50% "$file" "`basename "$file" .jpg` (small).jpg" 
done

4. First, we install the software we need:

sudo apt-get install vorbis-tools
sudo apt-get install lame

Then we convert the files:

wget http://helms-deep.cable.nu/~rwh/files/lara_st_john.tar.gz
tar xvzf lara_st_john.tar.gz
cd lara_st_john/bach_violin_concertos
for file in *.ogg
do
        oggdec "$file"
        lame --vbr-new `basename "$file" .ogg`.wav `basename "$file" .ogg`.mp3
done

5. First, we extract the track information from the ogg file using the vorbiscomment command and save it in a temporary file. Then we use grep and sed to select each datum and save it in a shell variable. Once we have all this info ready, we can decode the ogg file to wav, then encode the wav to mp3, passing in the shell variables which contain the track information.

for file in *.ogg
do
        base=`basename "$file" .ogg`
        vorbiscomment "$file" > "$base.comment"
        tt=`grep -e "^title"  "$base.comment" | sed -e 's/title=\(.*\)$/\1/'`
        ta=`grep -e "^artist" "$base.comment" | sed -e 's/artist=\(.*\)$/\1/'`
        tg=`grep -e "^genre"  "$base.comment" | sed -e 's/genre=\(.*\)$/\1/'`
        ty=`grep -e "^date"   "$base.comment" | sed -e 's/date=\(.*\)$/\1/'`
        tl=`grep -e "^album"  "$base.comment" | sed -e 's/album=\(.*\)$/\1/'`
        oggdec "$file"
        lame --vbr-new --tt "$tt" --ta "$ta" --tg "$tg" --ty "$ty" --tl "$tl" "$base.wav" "$base.mp3"
        rm -f "$base.comment"
        rm -f "$base.wav"
done