Thursday, September 19, 2013

How to run multiple Google Notifiers for different GMail accounts in OS X

  1. Prepare app for re-signing

    1. Duplicate and rename the notifier .app file.
    2. Right-click on the copied .app and show package contents.
    3. Edit the files App.icns, GmailIcon.png, nounread.png, and unread.png in Content/Resources so that they can be distinguished from the original images (so you can tell which notifier is showing notifications).
      • For the png's, I opened each in Paintbrush.app and inverted the colors.
      • I replaced App.icns with the .icns file from another mail program that I don't use.
    4. Open the Info.plist file in Contents in a text editor and change the <string> associated with the CFBundleIdentifier key from com.google.GmailNotifier to something different but recognizable.
  2. Obtaining a Signing Identity Note: this is necessary to avoid being prompted for a password each time the program is started.

    1. Open Applications > Utilities > Keychain Access.
    2. From the Keychain Access menu, choose Certificate Assistant > Create a Certificate.
    3. Fill in a name for the certificate. This name appears in the Keychain Access utility as the name of the certificate.
    4. Choose Self Signed Root from the Identity Type popup menu.
    5. Choose Code Signing from the Certificate Type popup menu.
    6. Check the Let me override defaults checkbox. Click Continue twice.
    7. Specify a serial number for the certificate. Any number will do as long as you have no other certificate with the same name and serial number. Click Continue.
    8. Fill in the information for the certificate. Click Continue until you can click Done.
  3. Re-sign application We will use a codesign command-line tool to re-sign the notifier application. Below, use whatever name you have given to the duplicated notifier .app file place of "Google Notifier 2.app".

    1. /usr/bin/codesign -f -s "MyCertificate" "/Applications/Google Notifier 2.app"
    2. Click Allow: /Applications/Google Notifier 2.app: replacing existing signature
  4. References

    1. http://hints.macworld.com/article.php?story=20061117161341318
    2. http://web.archive.org/web/20110224034132/http://newmotiongear.com/blogs/index.php/2011/02/20/how-to-re-sign-google
    3. http://web.archive.org/web/20110224034132/http://developer.apple.com/library/mac/#documentation/Security/Conceptual/CodeSigningGuide/Procedures/Procedures.html

Wednesday, February 13, 2013

Spyder and IPython in OS X Mountain Lion

These notes describe how to use homebrew and pip to get Spyder and IPython working in OS X Mountain Lion.  They are based in part on this blog post.

Note

Install XCode (from App Store) and Command Line Utilities for XCode (from Apple website after filling out a bunch of information) before proceeding. These packages install a bunch of compilers not included in OS X by default.

Brew

* Overview homebrew is a software management system for OS X built on Ruby and git that allows people to write install scripts for various packages that handle all required dependencies and compiler flags in a controlled manner. It is similar to MacPorts but seems to work a little better, possibly because its user base has more active contributors who keep its formulae up to date and working.
Warning: brew is kind of pushy about keeping it updated. Be careful about running brew upgrade. Don't do it just to keep up to date. Do it when you need new packages for specific features or to install other packages that depend on them. I have found that formulae are often updated without thorough testing so sometimes upgrades can break things. Consider looking at recently opened issues on the github pages of taps you will be upgrading from to see if there are any open issues that might affect the upgrade.
* Installation
ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)" then add to .bash_profile: export PATH=/usr/local/bin:$PATH

Python

brew install python --framework --universal then add to .bash_profile: export PATH=/usr/local/share/python:$PATH and change link to Python: cd /System/Library/Frameworks/Python.framework/Versions sudo rm Current ln -s /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/Current
Not much to add to the commands above.... brew installs some extra dependencies like readline and sqlite and some extra packages that are needed for the steps below like disutils and pip (not really sure how it did this).

numpy and scipy

numpy and scipy provide python wrappers to linear algebra packages and additional compiled computational code. Online I have seen people have issues trying to install them with pip. I tried to install from source following the instructions from scipy and had moderate but not full success (really, full success for numpy, partial success for scipy). What ended up working best for me was installing using a third-party brew tap:
brew tap samueljohn/python
brew install numpy --HEAD
brew install scipy --HEAD
These commands installs the latest dev versions of numpy and scipy from github. They also include some dependencies like nose (python module), suite-sparse (brew formula), and swig (brew formula).
You can run numpy and scipy's unit tests to see if the installation went okay. I found that scipy's test generated tons of errors or warnings. Talking to the developers and other users, I learned that these were okay and were the result of the tests not being well written for builds using Apple's Accelerate framework for the linear algebra package. This package is the best choice because it is optimized for Apple's Intel processors. Building an optimized linear algebra package from source is highly non-trivial.
A better way to test the numpy/scipy installation is to run some code representing typical usage that you know works or to run the unit tests of a program that uses. For example, I trust that the errors I saw in scipy's unit tests are not critical because qutip ran all of its tests successfully (plus the developers also say the test results are not a problem).

matplotlib and ipython

* matplotlib and ipython For matplotlib and ipython, I just used pip: pip install matplotlib pip install ipython
To use all of the features of ipython, you need Qt and zmq.
* Qt I did this, but I don't think it's necessary because the brew pyqt formula installs qt on its own: (I downloaded the Qt library installer from . Only the library is necessary (previously I downloaded the SDK which puts a lot of extra stuff on your computer). I got the open source libraries -- not sure if there are other versions available.)
After Qt, you need PyQt which I got by brew install pyqt and then adding to .bash_profile: export PYTHONPATH=/usr/local/lib/python:$PYTHONPATH
The PyQt formula adds some dependencies like sip and libpng.
* zmq Again I used brew: brew install zmq For the rest of the dependencies I used I used pip: pip install pyzmq pip install pygments pip install tornado
* ipython usage To use ipython as an interactive shell with function documentation popups and syntax highlighting, do: ipython qtconsole To enable plots to appear within the shell output rather than in separate windows, use the qtconsole with the pylab inline option: ipython qtconsole --pylab=inline To use ipython as a browser based notebook that can be saved and reopened, use ipython notebook To use a non-default browser like safari or firefox, use the --browser option: ipython notebook --browser=firefox For some reason, Google Chrome is not recognized by python. For it you have to use: BROWSER=/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome ipython notebook or (if you don't want ipython to open a new tab because Chrome will keep the tab open from last time): ((sleep 1 && /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome ) &) ; (ipython notebook --no-browser) In the latter, the sleep command gives ipython a chance to boot up its server before Chrome can load and say the page was not found. The command after a && is not executed until the command before it terminates successfully. bash runs commands surrounded by "(" and "&)" as background processes and executes commands separated by ";" simultaneously.

Spyder

Spyder has a few more recommended modules that were not installed above. I installed all of them via pip: pyflakes, rope, sphinx, pylint, and pep8. Spyder itself can also be installed via pip.

wxPython

I installed wxPython with brew install wxPython.

Fuse for OSX, Bitcasa, and encfs

This post describes installing encfs in OS X Mountain Lion 10.8.2 for use with Bitcasa (or any other remote file storage device). encfs is a really convenient file encryption tool. It allows you to mount a folder on your computer as a drive. When you move files into this drive, they are saved into the folder in an encrypted format, though within the mounted drive they can be opened normally. When the drive is unmounted, only the encrypted files remain. You can, for example, mount a shared folder (like a DropBox folder) and save files into in an encrypted form. If someone else ever gains access to your DropBox folder, they would still need the encryption key to do anything useful with your files. See e.g. this tutorial for some more specifics on how to work with encfs.

FUSE (Filesystem in user space) is a Unix module that lets programs create their own file systems on top of whatever systems the actual operating system is using. FUSE support in OSX is patchy with a previous project, macfuse, now defunct and several independent successors now being supported, mainly osxfuse and fuse4x.

As of early 2013, brew only supports fuse4x. According to the brew developers, "We would love to switch to OSXFUSE but unfortunately the buildsystem is extremely difficult to work with for producing the kind of installation we need." Bitcasa silently installs osxfuse along with its client (Bitcasa's install does not include osxfuse's prefpane for System Preferences). According to the brew developers, fuse4x and osxfuse use similar/the same names for their libraries, so having both installed can cause problems for programs trying to use one of them, especially since osxfuse installs into /usr/local where brew also puts its packages. It is advisable not to install both.

Both Bitcasa and encfs require fuse support. Since I want to use Bitcasa, I decided not to use fuse4x or the encfs brew formula that depends on it. Instead I did the following.

  1. Uninstall the Bitcasa client. I used the uninstall script within Bitcasa.app's package contents, though this ran into trouble when it got to the part that removes osxfuse and macfuse....

  2. git clone https://github.com/jollyjinx/encfs.macosx and run install.sh which downloads and runs the osxfuse installer and then installs encfs configured for osxfuse.

  3. Open osxfuse's pane in System Preferences and check the version.

  4. Reinstall Bitcasa.

  5. Check osxfuse's version again. Bitcasa had installed an older version on top of the one just installed, so I updated osxfuse back to the latest version.

After these steps, both Bitcasa and encfs worked for me. Most of the work involved here was done by Patrick Stein in creating the brew formula for encfs that uses osxfuse. If you want to follow the steps above, be sure to read over `install.sh` and `encfsmacosxfuse.rb` and make sure they look okay. I don't know how often the formula will be updated over time to keep it functional. As noted in the encfs.macosx usage notes, encfs drives can be mounted in OS X with, e.g.:

encfs ~/Dropbox/Documents.encfs ~/Dropbox/Documents.secure -- -o volname="Documents"

By default, encfs stores the encryption key for an encrypted folder in the .encfs6.xml file in the folder. If you don't want to keep the key in this folder (e.g. you want to save the encrypted files on a shared drive where you wouldn't want people to have access to the key), you can store the key somewhere else and then run encfs with the ENCFS6_CONGFIG flag like so:

ENCFS6_CONFIG="/Users/username/.keys/.encfs6.xml" encfs ~/Dropbox/Documents.encfs ~/Dropbox/Documents.secure -- -o volname="Documents"

After installing osxfuse, brew doctor will complain about the extra files that were installed into /usr/local. Unfortunately, you just have to learn to live with this. I don't understand why brew doesn't at least support osxfuse as a bottle (meaning that it just runs the binary installer rather than building the files itself), but I am sure there are good reasons. Here is what I get for brew doctor after doing the above steps:

Warning: Unbrewed dylibs were found in /usr/local/lib. If you didn't put them there on purpose they could cause problems when building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
    /usr/local/lib/libmacfuse_i32.2.dylib
    /usr/local/lib/libmacfuse_i64.2.dylib
    /usr/local/lib/libosxfuse_i32.2.dylib
    /usr/local/lib/libosxfuse_i64.2.dylib
Warning: Unbrewed .la files were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .la files:
    /usr/local/lib/libosxfuse_i32.la
    /usr/local/lib/libosxfuse_i64.la
Warning: Unbrewed .pc files were found in /usr/local/lib/pkgconfig.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .pc files:
    /usr/local/lib/pkgconfig/osxfuse.pc

Wednesday, January 2, 2013

Setting up a Linux server for versioned backups with rsync

Backup notes

Introduction

These notes describe setting up a simple Ubuntu backup server for backing up remotely from another computer running Linux or OSX. It is probably also possible to follow these notes to backup to a computer running OSX or backup from a computer running Windows (via cygwin) (Some subtle adjustments would probably be necessary. For instance, you might need to tell rsync to use less precision in comparing file modification times). These notes also assume that the backup server is being set up on a router, but it would probably not be too hard to set it up without one.

Choose back up disk file system

I think the best choice is just to go with ext3 since it is native to Linux. The downside is that it does not have as good cross-platform support as other systems such as FAT32 (FAT32 does not have the Unix permissions that rsync tries to copy with the -a flag. Also, FAT32 does not support symlinks and has reduced time resolution compared to other options), so it might be difficult to get a lot of files off the backup disk quickly (i.e. you couldn't plug the disk into a computer running OSX or Windows without installing a utility to translate from ext3). Here are some resources as of 1/1/2013 for accessing ext3:

Windows http://www.ext2fsd.com/

OSX http://reviews.cnet.com/8301-13727_7-57457850-263/how-to-manage-ext2-ext3-disks-in-os-x/ http://osxfuse.github.com/ http://sourceforge.net/projects/fuse-ext2/

Set up Ubuntu

There are few steps here:

  • Install Ubuntu (http://www.ubuntu.com/download/desktop). It should be pretty straightforward to set up a USB installer following Ubuntu's directions. To be safe, choose a strong password for any user accounts that can sudo.

  • Create a user account that you want to use for logging in remotely (this could be the same as the account that you created when installing Ubuntu, just be aware of its login name and password).

  • Install the backup disk (if it is not the boot drive...). Assign a static name to the drive (i.e. don't just use the default name that Ubuntu gives it so it can't be unmounted/remounted and assigned a different name).

  • Create a directory on the backup disk to hold the backups and make sure that the backup user account has write access to the directory (using chmod).

  • Install an ssh server. To do this, just run this command:

sudo apt-get install openssh-server openssh-client

By default, the server is set to run on startup. So there is not much more to do, but see the section on remote access below.

Set up RSA key pair with ssh-keygen and ssh-copy-id

Allowing traditional login via a password is not that safe because an attacker could try to crack it via repeated logins. An alternative is to setup an RSA keypair between the remote computer to be backed up and the server. This is basically like using a really long password that is stored on the two computers, so no password needs to be entered each time a connection is made.

I was setting up a laptop running OSX as my remote machine, so I did the following:

  • ran ssh-keygen on the remote machine in the home directory.

  • installed ssh-copy-id with:

sudo /usr/bin/curl "http://hg.mindrot.org/openssh/raw-file/c746d1a70cfa/contrib/ssh-copy-id" -o /usr/bin/ssh-copy-id

(I had to use /usr/bin/curl because another piece of software messed up my path to curl).

  • ran ssh-copy-id user@server to copy the RSA key to the server. Here user is the login name to use on the server and server is the ip address of the server. We haven't set up remote access yet, so the remote machine needs to be connected to the same router as the server and the ip address is the local ip address on the router (probably 192.168.1.x for some number x -- on most routers, you can check the ip address by logging into at with a web browser at 192.168.1.1). You should be prompted to enter the user account's password.

Set up remote access

Now we are almost ready to open up the server to the outside world.

Before exposing the server to the outside, we review a few notes about security. Security concerns are addressed here (I posted the question):

http://askubuntu.com/questions/229944/how-to-set-up-an-rsync-backup-to-ubuntu-securely

Basically, set PasswordAuthentication no in /etc/sshd_config and make sure that challengeresponse is also set to no. If you don't have a router, you should turn on a firewall and consider fail2ban.

To open the server to the outside world, set up the router to forward TCP on port 22 (the default port for ssh) to the server. On my Actiontec router from Verizon, this was simple to do (login into the router at 192.168.1.1, go to "Firewall Settings"->"Port forwarding", and then select the server in the first drop-down menu and SSH in the second). On other routers, it might be necessary to enter the server's MAC address and assign it a static ip address on the router and then forward the port to that static address.

Now the server can be accessed via ssh from the outside world. To do so, you need the ip address of the router. The ip address can be checked in various ways, for example by going to http://whatismyip.org/. Most home internet connections have dynamic ip addresses that can change. To create a static address that will always work, you can use a dynamic DNS service like http://www.no-ip.com/ (a good free one). The Actiontec router is set up to work with several dynamic DNS services (including no-ip; just go to "Advanced"->"Dynamic DNS" in the router web interface), so I just set it up to update no-ip when its ip address changes. Alternatively, one could set up a client like ddclient (http://sourceforge.net/apps/trac/ddclient/) on the server to monitor the ip address and update the dynamic DNS service.

A lot of the dynamic DNS services have cut back on their free options, so it might be necessary to hunt around for one (http://freedns.afraid.org is another one that looked promising but wasn't a default option in my router). I had set up a script to check my ip address and upload it to an ftp server before the respondent to my askubuntu.com question said to just go with the simplest solution. Something like that is an option though. All you need is something that will check the ip address (by going to e.g. and parsing the content) and then send something out somewhere when a change is noticed. So far, I have left my router on and plugged in and the ip address has not changed in over a month, so you could get away without setting up anything as long as you were okay with the occasional break in connectivity (dependent upon how often your ip address changes).

Backup script setup

Finally, a script needs to be written to handle the transferring and managing of backups from the remote machine to the server. I used the following two scripts as my starting point:

http://randombytes.org/backups.html http://www.jan-muennich.de/linux-backups-time-machine-rsyn

The rsync command does most of the work. The rest of the scripts are basically just error checking the connection and previously started backups and then deleting old backups. You can set the script to run automatically with something like cron or Automator or you can just run it by hand. I like creating single line AppleScript applications that call other scripts with "do shell script" so that I can run them from SpotLight.

My current scripts are wsbackup_transfer.sh and wsbackup_prune.py available at https://github.com/willsALMANJ/wsbackup.

The generic entries in my excludes file are: .DSStore Thumbs.db *~ .*

Appendix: extra stuff not done

IP update through ftp instead of dynamic DNS

There are some free services available for reserving and updating a dynamic DNS address to point to your home ip. When I looked into this, most sites recommended DynDNS (of dyn.com) as the premier option that many manufacturers built in support for (stuff like webcams and media streaming devices). However, Dyn had just decided to discontinue its free service except for allowing one free address that you could get by signing up for their pay service and then canceling during the 14-day free trial. The free address came with a 30-day inactivity expiration. There are other sites like no-ip.com and freedns.afraid.org that still have more generally free services, but it seems like in general free dynamic DNS services are being phased out by (pretty cheap) pay services.

An alternative that I thought of was to use an ftp account I have access to to push a file containing the server's ip address to and then to grab that file with the remote computer before connecting with rsync/ssh. Here are the commands for scripting these sets of actions. I included an encryption of the ip address just to draw less attention since the ftp transfer was not secure.

Get the current ip address and write to file: curl -s checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//' > ip

Encrypt file with ip address: openssl aes-256-cbc -salt -a -e -pass pass:password -in ip -out ipenc

Push file to ftp account: /usr/bin/curl -Q '-SITE CHMOD 600 ip' -u user:password -T ip ftp://ftp.domain.com

Get file from ftp account: /usr/bin/curl -d user:password -T ip ftp://ftp.domain.com

Decrypt file: openssl aes-256-cbc -salt -a -d -pass pass:password -in ipenc -out ip

Other safety measures

These safety measures were discussed at some places but are for more public servers:

  • Use a firewall on the server (ufw)
  • Use a non-standard port for the transfer
  • Use fail2ban

Some of these suggestions are discussed in the references below:

http://askubuntu.com/questions/85462/what-is-my-computer-ip-address-knowing-that-i-have-a-router http://www.cyberciti.biz/faq/unix-linux-bsd-osx-change-rsync-port-number/ http://www.howtogeek.com/115116/how-to-configure-ubuntus-built-in-firewall/ http://hints.macworld.com/article.php?story=20031024013757927 http://www.fail2ban.org/wiki/index.php/Main_Page

Wednesday, November 14, 2012

iTunes + iPod Setup

These are the settings I use to listen to Podcasts efficiently with iTunes.

Podcasts set up

First off, I subscribe to a bunch of podcasts in iTunes. I make the default settings for podcasts download the most recent one and keep all unplayed episodes. For some really active podcasts, I have the download option set to do nothing, and then I download the episodes I want by hand.

Podcast playlist set up

Basic setup

Next, I create a smart playlist that searches for all items with Media Kind of "Podcast" and play count of 0. With this set up, new podcasts (if set to download the latest episode) will automatically appear at the end of this playlist and old podcasts will disappear as soon as they finish playing. iTunes will eventually delete the played podcasts automatically as well (if "episodes to keep" is set to all unplayed episodes).

I like the above set up. You don't have to worry about managing the podcast files at all (except for choosing episodes to download from podcasts that you don't have set to automatically download the latest episode).

Adding non-podcast files to playlist

Sometimes I need to add non-podcast files to the podcast playlist (like podcasts that are not posted to iTunes and have to be downloaded manually). To do this, I open the file in iTunes and then right-click on it (in iTunes) and choose Get Info->Options and set the Media Kind to Podcast. Doing this causes the file to disappear from the "Music" section of iTunes and to appear in the Podcast smart playlist. It can now be sync'ed to the iPod normally. After it is played, it will disappear from the playlist though. To make sure that it is really removed from iTunes, I create another smart playlist called "ZAllPodcast" (the "Z" is there to make the playlist be on the bottom of the list since I never listen to this playlist) that includes all files with Media Kind = Podcast and no other conditions. The played podcast will still appear here. To delete from iTunes, select it and then do Option+Delete. You also have to delete the audio file by hand.

iPod sync set up

For music, I choose to sync only selected playlists, artists, albums, and genres because I have a large music library. For podcasts (and iTunesU), I sync all unplayed episodes of all podcasts.

Syncing the iPod

This is the tricky part. I have an AppleScript called SmartIpodSync.scpt that I use to eject the iPod. Here are the steps of the script (the script is available at https://github.com/willsALMANJ/applescripts):

  1. Check for iPods and exit if none are found. Otherwise perform the following steps for each iPod.
  2. Check that UIScripting is enabled in System Preferences and show a message explaining how to enable it if it is not. (This is necessary because the script needs to set several properties not exposed to AppleScript and interacts with them by telling iTunes which UI elements to click on).
  3. Find all played podcasts and tell both iTunes and Finder to delete them. This is done in case podcasts were downloaded and listened to on the computer but not yet cleaned out by iTunes. By deleting these podcasts now, iTunes will not waste time sync'ing them to the iPod later.
  4. Using UIScripting, turn off live updating of all smart playlists sync'ed to the iPod. This step is necessary because when iTunes sync's a smart playlist with live updating on the playlist appears in the iPod with an unexpected ordering (maybe date downloaded?). I like to actively keep rearranging my podcasts by hand and then just always play the top one on the playlist (which disappears as soon as it is played). That way stuff floats to the top as I listen (since I mostly listen chronologically with small tweaks).
  5. Sync the iPod. Using UIScripting, monitor the text of the status window at the top of iTunes for "iPod sync is complete." and allow the script to continue once this text appears. This waiting loop allows the iPod to sync before continuing the script without waiting extra time.
  6. Eject the iPod from iTunes.
  7. Using UIScripting, go back through the smart playlists that were sync'ed and turn live updating back on so that new podcasts (and songs for other playlists) will be added automatically.
  8. Tell Finder to hide iTunes (set visible of process "iTunes" to false).

Thunderbird-GMail set up and usage notes

Introduction

Motivation

GMail's web interface is really nice, but I like having a local copy of my email on my computer. Having a local copy managed by an email program is useful for a few different reasons.

  1. For reference when there is no internet access. With the rise of smartphones, the likelihood of someone being cut off from email has been greatly diminished. Personally though I still do not own a smart phone. Also, occasionally it is beneficial to have a copies of attachment files on your actual computer, and they might be hard to transfer from your phone if you are temporarily without email access.

  2. For backup. Google is a pretty successful company and is probably not going away soon, but it is still possible that your account could get hacked or otherwise corrupted. Having a local copy of your email that you can backup to additional locations provides added redundancy.

  3. Thunderbird gives greater control over some aspects of account management than GMail does. For instance, it is possible to delete individual attachments from messages. In GMail, one can only delete the messages themselves. There are many Add-ons for Thunderbird, and some of them might be useful for account management. For instance, the Delete Duplicate Messages* extension makes it easy to identify duplicate messages for deletion. It is also not that hard to write simple extensions of your own.

Usage Modes

I use Thunderbird and GMail for three different kinds of email accounts: my main personal email account (a GMail account), my work email (a non-GMail IMAP account), and my old email from defunct email accounts (eg my college email). The regular GMail is fairly straightforward. Interfacing it with Thunderbird will be discussed below. I keep all the mail from old email accounts in the "Local Folders" section of Thunderbird.* I wanted to briefly outline the setup for my non-GMail IMAP account here since it is very similar to the GMail account but has some small differences that will be pointed out below.

The IMAP account I use is associated with an Exchange web mail interface, which I do not like. In order to avoid using this, I switched it to a GMail account "by hand." The switch required:

  1. Creating a new GMail account to associate with this other email account.
  2. Setting the original account to forward all mail to the new GMail account without saving a local copy (so that I never have to worry about the original account using up all of its storage space).
  3. Setting the GMail account to use the original account's SMTP server, so that outgoing mail appears to come from the original account's address (in GMail: Settings->Accounts and Import->Send mail as; see this Google support page for more).

Thunderbird Setup

Basic setup

The basic Thunderbird setup is covered in various blog posts and support sites online (such as this MozillaZine one). Looking things up online might not even be necessary. If you just do, Tools->Account Settings->Account Actions->Add a mail account and then enter your GMail account information, you get the basic set up done almost automatically. The only choice is IMAP vs. POP3. I use IMAP, just because it is what I have always used.

Folders versus Labels

It is important to understand how Thunderbird/IMAP and GMail treat folders/labels differently. In GMail, there is basically just "All Mail" and then labels that get added to that mail. You can add as many labels as you would like. GMail still keeps just one copy of that email in your "All Mail" "folder". IMAP organizes mail into folders with individual emails acting like the files of a traditional file system. GMail translates its labels into IMAP folders when it syncs with an email client via IMAP. If you like to use a lot of labels, you will end up with a lot of duplicated messages in your IMAP files -- one for each GMail label that a message has (see "Folder Subscriptions" below for a potential solution).

Message Archiving Strategy

My email archiving strategy is to keep one copy of each message. Additionally, I try to limit the size of individual email folders to a few hundred MB's if possible and to keep most of my old mail in inactive folders. Thunderbird stores the contents of an IMAP folder as a single file. If you use a versioned data backup system, every time anything changes in that folder that single file is modified and the new version is saved to the backup system. It saves backup transfer time and storage space to keep most old emails in folders that are rarely modified and thus do not require new backups constantly. I don't worry about small folders, but for folders that cover my largest amount of space (eg correspondence with family) I make an "archive" version of that folder in which I create a separate subfolder for each year. Then I save all messages from 2010 to the 2010 folder, all from 2011 to the 2011 folder, etc. Keep in mind that all sent mail goes to the Sent Mail folder by default -- you might want to move old sent mail out of the Sent Mail folder and into archives to keep the Sent Mail folder size down.

Folder Subscriptions

Thunderbird allows you to subscribe to the folders (ie the GMail labels) in your IMAP account individually. These subscriptions can be managed by clicking on the account in Thunderbird and then "Manage folder subscriptions." By default all folder subscriptions are turned on.

I recommend unsubscribing from these subfolders of the [GMail] folder: All Mail, Important, and Starred. Each of these folders will automatically duplicate messages saved in other folders, so if you are following my strategy above of storing all messages in well defined folders you will end up duplicate messages in your Thunderbird account.

If you like to use multiple GMail labels for individual emails, you will also end up with duplicate messages for individual emails. One solution to this problem would be to create separate folders for Thunderbird, for example just a single folder for each year, and then copy all messages to the appropriate folder (or label if you do it in GMail). Then you could keep your old labels in GMail and just subscribe to the new labels in Thunderbird (with the extra burden being sure to add these new labels to your new messages in GMail in addition to your old labels).

A couple additional notes about setting up folders and avoiding duplicate messages:

  • The "All Mail" folder in GMail contains all received mail. All of your mail (not counting what is in Spam or Trash -- they are not visible in All Mail in GMail but might actually be there hidden somehow) should be in All Mail.

  • The "Sent Mail" folder is similar to the "All Mail" folder but contains only mail that you have sent. Like the "All Mail" label, the "Sent Mail" label is hidden in the GMail interface. I subscribe to the "Sent Mail" folder because new messages that I compose have no other label. For a mail account that is used primarily via GMail, over time sent messages will get labeled when someone responds and I move the conversation out of my inbox by giving it a new label. At the end of the year, I quickly scan through my "Sent Mail" folder on GMail and label any messages that I sent that didn't get labeled during the year. Then I delete all of those messages from the Sent Mail folder in Thunderbird to remove the "Sent Mail" label. For a mail account used primarily with Thunderbird, the sent mail will only be in the sent mail folder, so I just move all of that to another folder at the end of the year.

  • When you assign a label to a message in GMail, it gives that label to all messages in the conversation. There is no way to label only part of a conversation.

  • If you are organizing messages by year, it is easier to copy messages to folders in Thunderbird. If you use GMail, any conversation that has messages in multiple years will end up with the same messages duplicated in the folders for each of those years.

  • If you are worried about duplicated messages, the Remove Duplicate Messages Add-on for Thunderbird can help you identify them (I recommend removing them by hand rather than trying to use the add-on to remove the messages because of the interaction between the way the add-on deletes messages and the way GMail treats deleted messages (see the Trash section below)).

  • One thing that is nice about dealing with duplicate messages: if you move a message from one folder in Thunderbird into another folder with the same message already in it, you end up with only one copy of the message in the folder (because both messages correspond to the same message in GMail and GMail is really just changing the message labels).

  • If you are worried about breaking mail up into individual folders making it difficult to read through your mail in a logical way, note that Thunderbird has an "Open in Conversation" function that lets you open a message in a separate tab with a tree view of all replies/forwards/etc related to that message -- similar to GMail's conversation view.

Trash

Like the rest of folders in GMail, the Trash is just another label. Thirty days after items are given the Trash label they are expunged from GMail. When you follow the default GMail setup in Thunderbird, messages are moved to the Trash folder in when you delete them (this can be set in the Server Settings section of the Account Settings for the email account). If you follow the message archiving strategy outlined above (ie no duplicate messages), this setting will cause things to work the way you expect them to -- the message will disappear from the folder it was in and appear in the Trash folder, and after thirty days it will be expunged.

With delete set to move a message to the Trash folder, it is important to keep in mind that if the message also had other labels and so was in other folders the copies of the message in those other folders will also be expunged after thirty days because in GMail there is really only one copy of the message that is being given different labels. If you just want to remove a message from a folder but do not want to remove it from GMail, you can switch the delete action to "Remove it immediately" and then delete the message. This method will never completely remove the message from GMail because the copy in All Mail is never removed until you move it to Trash and then expunge it from the Trash. If you move or delete things from All Mail, the copy in GMail will still be there (though it might take a while for Thunderbird to realize to re-sync and show the message in All Mail again). The "Remove it immediately" option is useful for eliminating duplicated messages (if you have the "Move to Trash" setting on, you will end up deleting all copies of the duplicated message!).

Sent Mail

The other setting that might need to be tweaked is the outgoing mail server. If you are connecting Thunderbird to a GMail account, then the outgoing mail server should be set up correctly with the default settings. However, if you are using GMail as the web interface for another account (as described in the "Usage Modes" section of the introduction), you will want to configure Thunderbird to use the original email account's SMTP server, so that mail you send appears to come from youraddress@originalaccount.com rather than yourgmailaddress@gmail.com. You can add SMTP servers by scrolling down to the bottom of the Account Settings window. The outgoing server for a particular email account can be selected in the top section of its of account settings (not one of the subcategories). See this mozillaZine article) for more.

Fine Tuning

With the basic, structural setup done, there are a few tweaks that can be implemented by using add-ons to closely mirror the behavior of GMail with Thunderbird. Here I mention a few extensions that I have found useful:

Keyconfig

This add-on works on all Mozilla programs (Thunderbird, Firefox, etc.). It provides a comprehensive interface for modifying existing keyboard shortcuts or creating new ones. In Thunderbird, I use it to copy GMail's keyboard shortcuts into Thunderbird, mainly J, K, C, O, F, #, R, A, and U. All of these mappings are straightforward renaming or copying of existing Thunderbird keyboard shortcuts except for U. To have U check for new mail when in folder view and to close the current message, return to folder view and check for new mail when looking at a message (in a separate tab from the folder view), I use the following code in Keyconfig:

if (((document.activeElement.id == 'messagepane') || 
    (document.activeElement.id == 'threadTree')) && 
    (document.getElementById('tabmail').tabContainer.selectedIndex != 0)) {
    CloseTabOrWindow();
}

goDoCommand('cmd_getMsgsForAuthAccounts');

One other note about standard Thunderbird keyboard shortcuts: * and \ expand and collapse all threads. This is useful to know because it is hard to navigate messages using only the keyboard when all threads are collapsed. On OSX, I find it hard to deal with some dialog windows from the keyboard. To choose "Don't Save" from the dialog that appears when closing a compose window, I have found that ctrl+n works.

Nostalgy

Nostalgy is a useful extension that allows you mimic some of the keyboard shortcuts used with labels in Gmail. In particular, I use Nostalgy's "Go to folder" function for GMail's jumping function (the G plus additional key set of shortcuts) and Nostalgy's "Save message" function for GMail's open move to function (the V shortcut). Nostalgy autocompletes folder names making it easy to move messages around (ie move them out of the inbox to an appropriate folder) and to navigate between folders with the keyboard. It has some additional features that are worth checking out as well (like a function to toggle the folder pane's visibility). Looking at the usage notes, I see lots of useful keyboard commands that I either just forget to use (eg pane/message navigation) or haven't taken the time to set up (eg rules for automatic message archiving).

QuoteCollapse

QuoteCollapse replaces quoted text in email messages with a small "+" icon that can be clicked on to show the quoted text (in Thunderbird, all quoted text is shown otherwise). It works pretty well, except that the icon is small and hard to click. Also, supposedly there is a keyboard shortcut to expand/collapse quoted text, but it did not work for me. Both of these issues were dealt with by creating a new expand/collapse text keyboard shortcut in Keyconfig. I use:

tree =  QuoteCollapse._messagePane.contentDocument.getElementsByTagName("blockquote");

if (tree.item(0).getAttribute("qctoggled") == "true") {
    QuoteCollapse._setTree(QuoteCollapse._messagePane.contentDocument, 0);
}
else {
    QuoteCollapse._setTree(QuoteCollapse._messagePane.contentDocument, 1);
}

which allows a single shortcut to toggle the expansion/collapse of quotes. To use separate keys for quote expansion and collapse, set the expansion shortcut to execute:

QuoteCollapse._setTree(QuoteCollapse._messagePane.contentDocument, 1);

and the collapse shortcut to execute:

QuoteCollapse._setTree(QuoteCollapse._messagePane.contentDocument, 0);

Lightning

Lightning adds a calendar feature to Thunderbird similar to Google Calendar (and other similar programs...). It can also sync to an existing calendar including a Google Calendar. If you search for how to sync Lightning with Google Calendar, you will find a lot of sites instructing you to use the Provider for Google Calendar add-on. This add-on works but is no longer necessary. As explained on this Google support page, it is now possible to sync Lightning to Google Calendar directly using CalDAV.

Summarizing those instructions: you use the New Calendar wizard to add a CalDAV network calendar and enter [ https://www.google.com/calendar/dav/ [ your Google Calendar ID ] /events ] for the location. Your Google Calendar ID for your primary calendar is just your email address, but for all other calendars the Calendar ID will be in the form of [ long string of characters]@group.calendar.google.com. You can get the Calendar ID by clicking the down arrow next to your calendar at calendar.google.com and selecting 'Calendar Settings'. The username is the full email address associated with that Google Calendar (and the password is that account's password).

Zindus

Zindus allows you to sync your GMail and Thunderbird contacts. Sync'ing physical addresses is a little tricky because GMail uses a single field and Thunderbird uses separate fields for street, city, zip code, etc. I rarely use contacts (I probably should make some new keyboard shortcuts to make it easy to add email senders as new contacts) and Zindus seemed to be constantly running, so I turned off auto-sync and just do a manual sync when I change a contact.

Remove Duplicate Messages

This add-on was discussed above. I just list it here for completeness. It is useful for identifying duplicated messages but not for actually deleting them (because it moves the extra copies to the Trash rather than just deleting them and causes GMail to remove all copies of the message).

Thunderbird Conversations

Thunderbird Conversations is an interesting add-on with a lot of appealing features. The short version is that skins Thunderbird to look and function almost identically to GMail. I wish it were more customizable because it has some features that I would like to use. However, I currently do not use it because those features come along with other features that I do not like and can not disable.

Here are the features I like:

  • Quick reply in the message pane. One of things that annoys me most about Thunderbird is that message composition must be done in a separate window. Thunderbird Conversations allows you to compose quick replies in the same window as the original message (like a reply in GMail). It also supposedly lets you compose new messages in a separate tab rather than a separate window, but I couldn't get that to work.

  • Attachment preview. Thunderbird Conversations shows you thumbnails of image and pdf attachments at the bottom of the image and can open them in a new tab (like GMail). Base Thunderbird can show image attachments if you toggle on View->Display Attachments Inline, but it is nice to be able to view pdf's as well.

Here is what I don't like:

  • The conversation view. Thunderbird Conversations replaces the message display window with a conversation display that looks like a GMail conversation including the GMail style fonts and stacking of individual messages, the GMail style message icons and action links, and the GMail style avatars for message senders. Thunderbird already has a conversation view that is easy to open with a keyboard shortcut and that replaces the current folder view a tree showing the conversation and individual messages appearing in the message window. The conversation can be navigated with the keyboard in the same manner as a folder. With Thunderbird Conversations, the whole conversation is now compressed into the message window and the extra GMail stylings take up even more space, leaving little room for the text. Also, supposedly it is possible to navigate through the conversation with the keyboard, but it didn't work for me -- maybe there was a conflict with my custom keyboard shortcuts. I didn't try too hard to get it to work.

  • The bugginess. Thunderbird Conversations changes so much of Thunderbird's basic interface that it seems to run into a number of conflicts. In the documentation, there are a few common problems listed, and reviewers on [AMO][http://addons.mozilla.org] list some others. No fault to the developer, but I would worry that every update to Thunderbird would break functionality in some way just because of all the changes Thunderbird Conversations builds onto the interface. Also, because of all these changes, opening messages incurred a small lag for me, which was slightly annoying when navigating messages.

I'd love a Thunderbird add-on that would enable viewing of pdf's within Thunderbird (and potentially other document formats like text documents or spreadsheets as GMail does). An add-on that allows message composition within the main Thunderbird window (in the message frame or a new tab) would also be a big upgrade. For now though, I am holding off on Thunderbird Conversations and waiting for these features to be implemented as separate add-ons or core Thunderbird features.

Tuesday, June 28, 2011

Installing Spyder on OS X 10.6.7 (Snow Leopard) on a MacBook Pro Core i7 (64 bit)

Note: see http://notcomputing.blogspot.com/2013/02/spyder-and-ipython-in-os-x-mountain-lion.html for notes on installing Spyder and IPython in OS X Mountain Lion.  Those newer notes could probably also be adapted to work with earlier versions of OS X.  The steps in the new set of notes are definitely easier than those in this older version.

Installing Spyder on OS X 10.6.7 (Snow Leopard) on a MacBook Pro Core i7 (64 bit)

Spyder is MATLAB-like GUI for a number of scientific computing packages written in Python.

Here is a rough outline of what I did to get Spyder working (specifically: Spyder 2.0.12). Over the course of installing Spyder, I came across several helpful tutorials and forum discussions. The links for these are given at the bottom.

Steps

1. Install the academic version of the 64-bit Enthought Python Distribution (EPD64) for OS X (epd-7.0-2-macosx-x86_64 for me). This package is available from enthought.com, but it requires an academic email address in order to be sent the download link.

This distribution came with Python 2.7 and many Python packages including pyflakes, sphinx, numpy, scipy, matplotlib, and IPython. EPD installed Python in a somewhat non-obvious place for me:

/Library/Frameworks/EPD64.framework/Versions/7.0/lib/python2.7

2. Install GCC compilers

I didn't really do anything for this step because I had already done it earlier before trying to install Spyder, but I saw it listed on another tutorial so I include it here. My OS X installation disc came with Xcode (3.2.3) already on it in the "Optional Installs" folder. Xcode was not installed by default when I received the computer. I installed it myself from the disc. Xcode provides the necessary compilers. Supposedly, Xcode can also be downloaded for free from the Apple Developer Connection website, but I didn't figure out how to do that before realizing I already had access to Xcode on the installation disc.

3. Install QtSDK

Qt is a cross-platform application and user interface framework. It is used to power the Spyder interface.

I downloaded the QtSDK OS X installer (Qt_SDK_Mac64_offline_v1_1_2_en.dmg for me) from Nokia at qt.nokia.com/downloads. I then just ran the installer and left every setting with the default value.

4. Install SIP

SIP is a program/module that helps interface Python with C/C++ libraries. It is a dependency of PyQt.

I downloaded the current version of SIP for OS X (sip-4.12.3.tar.gz) from http://www.riverbankcomputing.co.uk/software/sip/download. I saved it in the directory ~/Library/Spyder/ within my user directory. I then went to this directory in the terminal and extracted the files:

tar -xf sip-4.12.3.tar.gz

At this point, I changed directories to the one created by tar and tried to configure SIP but encountered an error ("Error: SIP requires Python to be built as a framework"). Looking around on the web, I found a discussion that recommended editing siputils.py file in the extracted SIP directory to change "Python.framework" to "EPD64.framework". After this (in the same terminal window as before -- in the sip-4.12.3 folder) I entered the commands:

python configure.py -d /Library/Frameworks/EPD64.framework/Versions/7.0/lib/python2.7/site-packages
make
sudo make install

In the first command, the path following the "-d" flag was the location of the site-packages folder of EPD64's Python installation. This location was chosen because Python has access to it on the PYTHONPATH.

5. Install PyQt

PyQt is a set of Python modules that interfaces between Qt and Python.

I downloaded the current version of PyQt for OS X (PyQt-mac-gpl-4.8.4.tar.gz) from http://www.riverbankcomputing.co.uk/software/pyqt/download. As with SIP, I saved it in the directory ~/Library/Spyder/ within my user directory. I then went to this directory in the terminal and extracted the files:

tar -xf PyQt-mac-gpl-4.8.4.tar.gz

I then moved to the PyQt-mac-gpl-4.8.4 folder and tried to configure PyQt as I had done with SIP using:

python configure.py -d /Library/Frameworks/EPD64.framework/Versions/7.0/lib/python2.7/site-packages

but I encountered the following error:

"Error: Make sure you have a working Qt v4 qmake on your PATH or use the -q argument to explicitly specify a working Qt v4 qmake."

I searched around with Finder for qmake and found two copies of it.... I just picked one and tried it:

python configure.py -q /Users/willshanks/QtSDK/Desktop/Qt/473/gcc/bin/qmake -d/Library/Frameworks/EPD64.framework/Versions/7.0/lib/python2.7/site-packages

generating the error:

"Error: Qt has been built as static libraries so either the -g or -k argument
should be used."

Finally I tried:

python configure.py -q /Users/willshanks/QtSDK/Desktop/Qt/473/gcc/bin/qmake -d/Library/Frameworks/EPD64.framework/Versions/7.0/lib/python2.7/site-packages -g

which did the trick. I followed this up with:

make
sudo make install

These commands generated a ton output in the terminal window and took several minutes to complete.

6. Install rope and pylint

These two packages are recommended/optional for Spyder and not included in EPD64. I downloaded pylint (pylint-0.23.0.tar.gz) from http://pypi.python.org/pypi/pylint and rope (rope-0.9.3.tar.gz) from http://rope.sourceforge.net/ into my ~/Library/Spyder/ directory. As above, I extracted them with tar -xf and in each case ran

sudo python setup.py install

from within the extracted folder. PyLint also generated a ton of output to the terminal including some "Syntax errors" but appeared to work for the most part (I am not sure how to test for its functionality yet).

7. Install Spyder

Finally, I downloaded Spyder (spyder-2.0.12.zip) to my ~/Library/Spyder directory from http://code.google.com/p/spyderlib/downloads/list. I unzipped this and moved to the spyder-2.0.12 directory in the terminal and entered the command:

sudo python setup.py install

8. Create Spyder launcher shortcut

Spyder should now be able launchable simply by typing "spyder" into a terminal window. Alternately, you can create a click-able icon using AppleScript. I just put:

do shell script "/Library/Frameworks/EPD64.framework/Versions/7.0/bin/spyder"

into AppleScript Editor and saved the file as an app. This app could then be clicked to launch Spyder.

Because I am particular about these things, I wanted my Spyder app to have the Spyder icon. To make this the case, I opened the svg file ~/Library/Spyder/spyder-2.0.12/spyderlib/images/spyder.svg that came with the Spyder zip file in InkScape and exported it as a 512 pixel wide png file. I then dragged this png file into Icon Composer (which was located in /Developer/Applications/Utilities) and produced a .icns file. I then downloaded Icon Droplet from http://download.cnet.com/IconDroplet/3000-2193_4-82563.html and ran it on the .icns file (not sure if this was really necessary). I then renamed this file applet.icns and saved it in the Resources folder of the Spyder app. This produced the Preview icon for the Spyder app. I then opened "Get Info" on the Spyder app, highlighted the Preview icon in the upper left, and pressed Delete. The icon then changed to the new Spyder icon. I am not sure if all of these steps were necessary or if some of them could have been skipped. I tried various iterations of these steps but I did not restart OS X in between -- that might have been needed for the icon to be updated everywhere.

To do: I haven't tested all of Spyder's functionality so I am not positive that everything is set up properly. Additionally, it would be nice to find a way to run Spyder without opening to "black box" exe icons in the dock.

Links

General installation of PyQt and Spyder:

http://code.google.com/p/spyderlib/wiki/Installation
http://works13.com/blog/mac/howto-install-spyder-1-x-on-mac-os-x-with-64bit-python.htm
http://blog.oak-tree.us/index.php/2010/05/27/pyqt-snow-leopard
http://www.oak-tree.us/blog/index.php/2009/05/12/pyqt-mac

Modifying SIP installation for EPD64 framework:
http://comments.gmane.org/gmane.comp.python.epd.user/138

Suggestion to NOT modify anything for PyQt, just add the -q and -g flags as described above:
(when trying to debug the errors with PyQt, I initially saw suggestions of modifying the PyQt setup files in a similar fashion to the way the SIP file was modified)
https://mail.enthought.com/pipermail/epd-users/2011-March/000337.html

Creating custom icons in OS X:
http://paulstamatiou.com/how-to-create-an-os-x-icon-from-a-vector-graphic
http://paulstamatiou.com/how-to-quickie-change-os-x-icons
https://discussions.apple.com/thread/3069882?start=0&tstart=0