From Arch to Ubuntu

After the third time of having to spend hours to fix my arch installation after an upgrade I've decided my time is best spent elsewhere.  Because of these issues I've moved back to Ubuntu.  Ubuntu tends to have better software support and vmbuilder is nice.

I started with the Ubuntu mini x86_64 installer as my base install.  This allows me to get a more minimal installation without unity and other tools that I don't use.  I've then installed multiple KVM Ubuntu JeOS images on top of that via vmbuilder and run my software on those.  

Right now my setup consists of:

Windows Manager: awesome

Terminal Program: terminator

Clipboard manager: parcellite

Text expanding: autokey-gtk

Virtual Machines: KVM (with libvirt to manage them)

I use IOMMU to pass my wifi card to a Linux VM that controls the wireless connections using wicd.  I then have a Linux VM that connects to the wifi VM and uses it to connect to my OpenVPN server.  After that all my other VMs connect to the VPN VM in order to get to the internet.  This allows me to force traffic on specific VMs through the VPN.  If the VPN isn't connected then nothing on my internal network can get out to the internet.  It's a bit overkill, but I find it to be an interesting project.  I'll write more details about my setup throughout the next few weeks.

 

From Arch to Ubuntu

After the third time of having to spend hours to fix my arch installation after an upgrade I've decided my time is best spent elsewhere.  Because of these issues I've moved back to Ubuntu.  Ubuntu tends to have better software support and vmbuilder is nice.

I started with the Ubuntu mini x86_64 installer as my base install.  This allows me to get a more minimal installation without unity and other tools that I don't use.  I've then installed multiple KVM Ubuntu JeOS images on top of that via vmbuilder and run my software on those.  

Right now my setup consists of:

Windows Manager: awesome

Terminal Program: terminator

Clipboard manager: parcellite

Text expanding: autokey-gtk

Virtual Machines: KVM (with libvirt to manage them)

I use IOMMU to pass my wifi card to a Linux VM that controls the wireless connections using wicd.  I then have a Linux VM that connects to the wifi VM and uses it to connect to my OpenVPN server.  After that all my other VMs connect to the VPN VM in order to get to the internet.  This allows me to force traffic on specific VMs through the VPN.  If the VPN isn't connected then nothing on my internal network can get out to the internet.  It's a bit overkill, but I find it to be an interesting project.  I'll write more details about my setup throughout the next few weeks.

 

IPv6 — getaddrinfo() and bind() ordering with V6ONLY

Recently I ran into an issue that took me a while to sort out, and it is regarding inconsistent behaviour on various OS's with regards to IPv6 sockets (AF_INET61) and calling bind(2) after getting the results back from getaddrinfo(3).

A call to getaddrinfo() with the hints set to AF_UNSPEC in ai_family and AI_PASSIVE in ai_flags will return to us 1 or more results that we can bind() to. Sample code for that looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
struct addrinfo hints, *addrlist;

memset(&hints, 0, sizeof(hints));

// Ask for TCP
hints.ai_socktype = SOCK_STREAM;

// Any family works for us ...
hints.ai_family = AF_UNSPEC;

// Set some hints
hints.ai_flags = 
            AI_PASSIVE    | // We want to use this with bind
            AI_ADDRCONFIG;  // Only return IPv4 or IPv6 if they are configured

int rv;

if ((rv = getaddrinfo(0, "7020", &hints, &addrlist)) != 0) {
    fprintf(stderr, "getaddrinfo: %s", gai_strerror(rv));
    return 1;
}

// Use the list in *addrlist
for (addr = addrlist; addr != 0; addr = addr->ai_next) {
    // use *addr as appropriate
}

// Clean up the memory from getaddrinfo()
freeaddrinfo(addrlist);

On Linux there are two entries returned when the host it is run on has both IPv4 and IPv6 enabled. An AF_INET which was followed by an AF_INET6. Now, it is not said that you are required to use all of the results that are returned, but if you want to listen on all address families it is off course suggested.

Following the steps below for each of the returned results should result in having 1 or more different sockets that are bound to a single port.

  1. Create the socket()
  2. Set any socket options you want (SO_REUSEADDR for example)
  3. Then bind() the socket
  4. After that call listen() (followed off course by accept() on the socket)

Only for some unknown reason (and errno is no help) bind() fails when you get to the AF_INET6, which was returned second. Searching online as to why the bind would fail doesn't give you any good results and the thing that is even worse is that if you run the same code on another platform such as FreeBSD, OpenIndiana or Mac OS X no such failure exists. However I started suspecting something was up when I started looking at the output from netstat -lan | grep 7020 on Mac OS X. Where 7020 is the port I passed into getaddrinfo().

tcp46      0      0  *.7020                 *.*  LISTEN     
tcp4       0      0  *.7020                 *.*  LISTEN

Wait a minute ... one of the sockets is on both IPv4 and on IPv6. Some more time spent searching the internet I came across RFC 3493 section 5.3, which is titled "IPV6_V6ONLY option for AF_INET6 Sockets".

As stated in section <3.7 Compatibility with IPv4 Nodes>, AF_INET6 sockets may be used for both IPv4 and IPv6 communications. Some applications may want to restrict their use of an AF_INET6 socket to IPv6 communications only.

This was going down the right route, so I changed my code so that in the steps listed above in number 2 I added the following code if the socket type is AF_INET6:

1
2
3
4
5
if (setsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY, &yes, sizeof(int)) == -1) {
    close(sockfd);
    fprintf(stderr, "setsockopt: %s IPV6_V6ONLY\n", strerror(errno));
    continue;
}

The RFC 3493 section 5.3 also states that this option should be turned off by default, which means that all IPv6 sockets can also communicate over IPv4. Thus technically setting the option manually in code the best way to fix the issue. FreeBSD has had this feature turned on (as in IPv6 sockets can only communicate with IPv6 and NOT IPv4) since 5.x.

The biggest issue is that the remaining operating systems (OS X and OpenIndiana) don't have the same behaviour as Linux which makes troubleshooting this issue more difficult than it should be. The issue is that the RFC doesn't specify what exactly the operating should do when it encounters a request to bind to the same port on IPv4 and IPv6. The only place where I have found this documented is in "IPv6 Network Programming" under "Tips in IPv6 Programming" chapter 4, section 4, appropriately titled "bind(2) Ordering and Conflicts".


If you get a bind() error when attempting to bind to an AF_INET6 socket please make sure that you set the socket option IPV6_V6ONLY on the AF_INET6 socket. The default as required by RFC 3493 is to have that option be off. The default is wrong, and the RFC should have been more specific regarding what the right behaviour is when attempting to bind on an AF_INET6 socket when already bound on an AF_INET while IPV6_V6ONLY is set to false.

The full code that I used for testing, along with a little bit more information is available as a gist on github.


  1. The old BSD style socket() called for defines starting with PF_ such as PF_INET and PF_INET6 with the PF standing for protocol family. POSIX starts them with AF_, and calls them an address family. On almost every operating system PF_INET is the same as AF_INET. If the define doesn't exist you can always create it. 

Converting KVM virtual machines to VirtualBox

Recently the requirement came up to take a KVM based virtual machine and move it over to a VirtualBox image. Which turned out to be a fairly simple endeavour, and was fairly painless. The longest part was transferring over the 40 GB image from one machine to the other where the conversion could take place. The machine the image was coming from was only on a 100 Mbit/sec connection so that took a good hour.

Converting from KVM to VirtualBox for a FreeBSD image was pretty simple, the VBoxManage command has a convertdd command that allows you to convert from raw disk .img format to .vdi format.

VBoxManage convertdd KVM-image.img VB-image.vdi

After this, unfortunately, there is no way to to automatically convert over the settings that the virtual machine had, such as the network cards, the memory allocations and hard drive settings. You will have to go to VirtualBox and create a new virtual machine and replicate all of the settings. Once that is done make sure to select the same type of disk controller (SATA or IDE) so that the drive will hopefully be assigned the same name in the device tree so that you don't need to alter your /etc/fstab.

Hopefully everything boots without any issues. If not try creating a new virtual machine, attach the converted image as a secondary drive and see if you can mount the converted image within your new install. If so maybe transferring the data using rsync or dump/restore would be an option.

My migration from OSX to Linux

I’ve been moving back and forth between OS X and Linux for the last 5 years or so, mostly due to performance reasons. Currently I’m moving away from OS X and back to Linux, hopefully for the last time. My main reason for this is to gain more control over what my computer does and what data it sends out. I no longer feel like I’m in control of my machine when I use OSX or Windows. However, I do still find OSX applications better from a productivity stand point. My main goal right now is to find good enough alternatives to the software I normally used in OSX. I’ll be making posts over the coming months on most of these programs and why I feel they’re useful to have.

Linux Alternatives for OSX Programs


OSX Program Linux Alternative(s) Reasons for Choosing Alternative
1password lastpass and keepassx I’m using two because lastpass has a good browser plugin so it’s useful for website information. Keepassx is more useful to keep other passwords, such as ssh account passwords.
Acorn GIMP Acorn is an image manipulation program. GIMP is the best Linux image manipulation program that I am aware of.
Adium Pidgin Pidgin isn’t as pretty as adium, but it works the same.
Hazel Custom Script Hazel can run scripts and move files on a regular basis based on filename. This should be able to be accomplished with shell scripts and a cron jobs.
iTunes Audacious I was a big fan of XMMS and Winamp, so I prefer something simple like Audacious over more of a music library manage like amarok.
Launchbar built-in awesome functions or Synapse The awesome windows manager has built in functionality for doing launches. Again, it’s not as nice as launchbar, but it works. Synapse also looks like an interesting alternative and I will be looking in the future to see how well that works in awesome.
NetNewsWire LifeRea LifeRea has a similar layout to NetNewsWire. One drawback is that LifeRea stores the google account password in cleartext. I’m accepting this risk by creating a separate google account that is only used for news feeds.
Notational Velocity ZIM or KeepNote Notational Velocity is a great note taking application that doesn’t require any explicit save option. I’m currently looking into ZIM and KeepNote as a replacement. I’ll be posting a blog post on this soon.
OmniFocus todotxt OmniFocus is hands down the best todo list program I’ve ever used. I haven’t found anything comparable for it in Linux. Instead I’m going back to the basics and using todotxt, which is a bash script that helps manage a text file todo list.
OmniOutliner vim A good outliner program that gets out of the way is hard to find. I haven’t found anything comparable in either Windows or Linux. Instead I’m using vim with some options to make it easier to deal with outlines.
OmniGraffle dia dia is the best diagram program that I’ve found for Linux.
Parallels VirtualBox VirtualBox lacks some of the features of parallels, such as encrypting VMs. However, it has other features that I use, such as VDE and IOMMU/VT-d support.
TextExpander AutoKey AutoKey is a python script that monitors the keyboard and will do a replacement of text when it sees a specific string. You can also have it run a python script and replace a string with the output of that script. I’m currently unsure of the security implications of using this program.
Time Machine deja-dup Deja-Dup is a GUI frontend for duplicity. I’m still in the process of determing if its better to backup to a portable drive or a file server.
Textual weechat I use weechat over irssi since I have an awesome plugin to monitor notifications in it.

Programs That I Haven’t Found Alternatives For


BusyCal - Calendar application.
Coderunner - Simple GUI text editor with color coding syntax.
Day One - Journaling Program
Fantastical - Enter calendar entries via text
Keyboard Maestro - Allows you to run macros via keyboard shortcuts.
Patterns - Used to test regexp strings.
Scrivener - Writing program designed for 10+ page documents. Unison - Usenet reader

My migration from OSX to Linux

I’ve been moving back and forth between OS X and Linux for the last 5 years or so, mostly due to performance reasons. Currently I’m moving away from OS X and back to Linux, hopefully for the last time. My main reason for this is to gain more control over what my computer does and what data it sends out. I no longer feel like I’m in control of my machine when I use OSX or Windows. However, I do still find OSX applications better from a productivity stand point. My main goal right now is to find good enough alternatives to the software I normally used in OSX. I’ll be making posts over the coming months on most of these programs and why I feel they’re useful to have.

Linux Alternatives for OSX Programs


OSX Program Linux Alternative(s) Reasons for Choosing Alternative
1password lastpass and keepassx I’m using two because lastpass has a good browser plugin so it’s useful for website information. Keepassx is more useful to keep other passwords, such as ssh account passwords.
Acorn GIMP Acorn is an image manipulation program. GIMP is the best Linux image manipulation program that I am aware of.
Adium Pidgin Pidgin isn’t as pretty as adium, but it works the same.
Hazel Custom Script Hazel can run scripts and move files on a regular basis based on filename. This should be able to be accomplished with shell scripts and a cron jobs.
iTunes Audacious I was a big fan of XMMS and Winamp, so I prefer something simple like Audacious over more of a music library manage like amarok.
Launchbar built-in awesome functions or Synapse The awesome windows manager has built in functionality for doing launches. Again, it’s not as nice as launchbar, but it works. Synapse also looks like an interesting alternative and I will be looking in the future to see how well that works in awesome.
NetNewsWire LifeRea LifeRea has a similar layout to NetNewsWire. One drawback is that LifeRea stores the google account password in cleartext. I’m accepting this risk by creating a separate google account that is only used for news feeds.
Notational Velocity ZIM or KeepNote Notational Velocity is a great note taking application that doesn’t require any explicit save option. I’m currently looking into ZIM and KeepNote as a replacement. I’ll be posting a blog post on this soon.
OmniFocus todotxt OmniFocus is hands down the best todo list program I’ve ever used. I haven’t found anything comparable for it in Linux. Instead I’m going back to the basics and using todotxt, which is a bash script that helps manage a text file todo list.
OmniOutliner vim A good outliner program that gets out of the way is hard to find. I haven’t found anything comparable in either Windows or Linux. Instead I’m using vim with some options to make it easier to deal with outlines.
OmniGraffle dia dia is the best diagram program that I’ve found for Linux.
Parallels VirtualBox VirtualBox lacks some of the features of parallels, such as encrypting VMs. However, it has other features that I use, such as VDE and IOMMU/VT-d support.
TextExpander AutoKey AutoKey is a python script that monitors the keyboard and will do a replacement of text when it sees a specific string. You can also have it run a python script and replace a string with the output of that script. I’m currently unsure of the security implications of using this program.
Time Machine deja-dup Deja-Dup is a GUI frontend for duplicity. I’m still in the process of determing if its better to backup to a portable drive or a file server.
Textual weechat I use weechat over irssi since I have an awesome plugin to monitor notifications in it.

Programs That I Haven’t Found Alternatives For


BusyCal - Calendar application.
Coderunner - Simple GUI text editor with color coding syntax.
Day One - Journaling Program
Fantastical - Enter calendar entries via text
Keyboard Maestro - Allows you to run macros via keyboard shortcuts.
Patterns - Used to test regexp strings.
Scrivener - Writing program designed for 10+ page documents. Unison - Usenet reader

Laptop Power Saving

Kernel Options

Back in August phoronix came out with an article about some kernel options for power savings on sandy bridge processors. In order to use these you can add the following to your kernel line:

pcie_aspm=force i915.i915_enable_rc6=1 i915.i915_enable_fbc=1 i915.lvds_downclock=1

pcie_aspm=force enables aspm power saving. There was a bug that was introduced in an earlier kernel which prevented aspm from working correctly. The fix for this was rolled into 3.2.5. There's also supposed to be another fix going into 3.3. I'm unsure if this option will be needed once 3.3 is released.

i915.i915_enable_rc6=1 enables an extra power savings mode for the GPU. On some machines this is said to cause artifacts. I have not experienced any issues with this, but I also don't have many tasks that require 3d support.

i915.i915_enable_fbc=1 enables frame buffer compression. This saves some video memory. I'm unsure how much this would actually save power wise.

i915.lvds_downclock=1 enables the kernel to lower the clockrate of the LCD panel. This should save a little power, but might cause flickering. Again, I have had no issues using this.

laptop-mode-tools

laptop-mode-tools is a set of scripts that are designed to save power when a laptop isn't plugged in. I suggest installing acpid and ethtool in order for these scripts to work correctly. You can add acpid and laptop-mode-tools to your DAEMONS=(..) section in the /etc/rc.conf file if they aren't already there.

laptop-mode-tools is mainly configured using the file /etc/laptop-mode/laptop-mode.conf. It's well documented, so I suggest going through and seeing if there's anything you feel you need to change. I mostly changed the options regarding low battery charge, increasing them a few %. This is more a personal preference, as it bothers me when my laptop drops below 10%.

There are other configuration files in /etc/laptop-mode/conf.d. This includes features such as auto hibernate, usb autosuspend, ethernet power savings, and other commonly used features. I'd suggest taking a look in that directory if you want to get a better idea of other areas you can tweak with laptop-mode-tools.

I did have an issue with my u24e shutting down on battery power with laptop-mode-tools enabled. Disabling ethernet power saving by editing /etc/laptop-mode/conf.d/ethernet.conf and changing CONTROL_ETHERNET="auto" to CONTROL_ETHERNET="0" fixed this issue.

module blacklisting

Some hardware will take a small amount of power if the kernel module for it is loaded. You can disable these by creating (or editing) the file /etc/modprobe.d/blacklist.conf and putting a list of modules you don't want loaded prefixed by the word blacklist. For instance, I don't use the webcam so I blacklist the driver:

# Disable webcam
blacklist uvcvideo

Powertop

There's also a good power diagnosis tool called powertop, which can be used to determine what's waking a computer and what other tweaks can be made. Be aware that powertop 1 hasn't been updated in awhile and gives a lot of false tips, such as disabling usb autosuspend when it's already disabled. Powertop 2 was being worked on and has some interesting features in it. I believe the best way to install this is to use powertop-git in the AUR.