Wednesday, February 13, 2013

Spyder and IPython in OS X Mountain Lion

These notes describe how to use homebrew and pip to get Spyder and IPython working in OS X Mountain Lion.  They are based in part on this blog post.

Note

Install XCode (from App Store) and Command Line Utilities for XCode (from Apple website after filling out a bunch of information) before proceeding. These packages install a bunch of compilers not included in OS X by default.

Brew

* Overview homebrew is a software management system for OS X built on Ruby and git that allows people to write install scripts for various packages that handle all required dependencies and compiler flags in a controlled manner. It is similar to MacPorts but seems to work a little better, possibly because its user base has more active contributors who keep its formulae up to date and working.
Warning: brew is kind of pushy about keeping it updated. Be careful about running brew upgrade. Don't do it just to keep up to date. Do it when you need new packages for specific features or to install other packages that depend on them. I have found that formulae are often updated without thorough testing so sometimes upgrades can break things. Consider looking at recently opened issues on the github pages of taps you will be upgrading from to see if there are any open issues that might affect the upgrade.
* Installation
ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)" then add to .bash_profile: export PATH=/usr/local/bin:$PATH

Python

brew install python --framework --universal then add to .bash_profile: export PATH=/usr/local/share/python:$PATH and change link to Python: cd /System/Library/Frameworks/Python.framework/Versions sudo rm Current ln -s /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/Current
Not much to add to the commands above.... brew installs some extra dependencies like readline and sqlite and some extra packages that are needed for the steps below like disutils and pip (not really sure how it did this).

numpy and scipy

numpy and scipy provide python wrappers to linear algebra packages and additional compiled computational code. Online I have seen people have issues trying to install them with pip. I tried to install from source following the instructions from scipy and had moderate but not full success (really, full success for numpy, partial success for scipy). What ended up working best for me was installing using a third-party brew tap:
brew tap samueljohn/python
brew install numpy --HEAD
brew install scipy --HEAD
These commands installs the latest dev versions of numpy and scipy from github. They also include some dependencies like nose (python module), suite-sparse (brew formula), and swig (brew formula).
You can run numpy and scipy's unit tests to see if the installation went okay. I found that scipy's test generated tons of errors or warnings. Talking to the developers and other users, I learned that these were okay and were the result of the tests not being well written for builds using Apple's Accelerate framework for the linear algebra package. This package is the best choice because it is optimized for Apple's Intel processors. Building an optimized linear algebra package from source is highly non-trivial.
A better way to test the numpy/scipy installation is to run some code representing typical usage that you know works or to run the unit tests of a program that uses. For example, I trust that the errors I saw in scipy's unit tests are not critical because qutip ran all of its tests successfully (plus the developers also say the test results are not a problem).

matplotlib and ipython

* matplotlib and ipython For matplotlib and ipython, I just used pip: pip install matplotlib pip install ipython
To use all of the features of ipython, you need Qt and zmq.
* Qt I did this, but I don't think it's necessary because the brew pyqt formula installs qt on its own: (I downloaded the Qt library installer from . Only the library is necessary (previously I downloaded the SDK which puts a lot of extra stuff on your computer). I got the open source libraries -- not sure if there are other versions available.)
After Qt, you need PyQt which I got by brew install pyqt and then adding to .bash_profile: export PYTHONPATH=/usr/local/lib/python:$PYTHONPATH
The PyQt formula adds some dependencies like sip and libpng.
* zmq Again I used brew: brew install zmq For the rest of the dependencies I used I used pip: pip install pyzmq pip install pygments pip install tornado
* ipython usage To use ipython as an interactive shell with function documentation popups and syntax highlighting, do: ipython qtconsole To enable plots to appear within the shell output rather than in separate windows, use the qtconsole with the pylab inline option: ipython qtconsole --pylab=inline To use ipython as a browser based notebook that can be saved and reopened, use ipython notebook To use a non-default browser like safari or firefox, use the --browser option: ipython notebook --browser=firefox For some reason, Google Chrome is not recognized by python. For it you have to use: BROWSER=/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome ipython notebook or (if you don't want ipython to open a new tab because Chrome will keep the tab open from last time): ((sleep 1 && /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome ) &) ; (ipython notebook --no-browser) In the latter, the sleep command gives ipython a chance to boot up its server before Chrome can load and say the page was not found. The command after a && is not executed until the command before it terminates successfully. bash runs commands surrounded by "(" and "&)" as background processes and executes commands separated by ";" simultaneously.

Spyder

Spyder has a few more recommended modules that were not installed above. I installed all of them via pip: pyflakes, rope, sphinx, pylint, and pep8. Spyder itself can also be installed via pip.

wxPython

I installed wxPython with brew install wxPython.

Fuse for OSX, Bitcasa, and encfs

This post describes installing encfs in OS X Mountain Lion 10.8.2 for use with Bitcasa (or any other remote file storage device). encfs is a really convenient file encryption tool. It allows you to mount a folder on your computer as a drive. When you move files into this drive, they are saved into the folder in an encrypted format, though within the mounted drive they can be opened normally. When the drive is unmounted, only the encrypted files remain. You can, for example, mount a shared folder (like a DropBox folder) and save files into in an encrypted form. If someone else ever gains access to your DropBox folder, they would still need the encryption key to do anything useful with your files. See e.g. this tutorial for some more specifics on how to work with encfs.

FUSE (Filesystem in user space) is a Unix module that lets programs create their own file systems on top of whatever systems the actual operating system is using. FUSE support in OSX is patchy with a previous project, macfuse, now defunct and several independent successors now being supported, mainly osxfuse and fuse4x.

As of early 2013, brew only supports fuse4x. According to the brew developers, "We would love to switch to OSXFUSE but unfortunately the buildsystem is extremely difficult to work with for producing the kind of installation we need." Bitcasa silently installs osxfuse along with its client (Bitcasa's install does not include osxfuse's prefpane for System Preferences). According to the brew developers, fuse4x and osxfuse use similar/the same names for their libraries, so having both installed can cause problems for programs trying to use one of them, especially since osxfuse installs into /usr/local where brew also puts its packages. It is advisable not to install both.

Both Bitcasa and encfs require fuse support. Since I want to use Bitcasa, I decided not to use fuse4x or the encfs brew formula that depends on it. Instead I did the following.

  1. Uninstall the Bitcasa client. I used the uninstall script within Bitcasa.app's package contents, though this ran into trouble when it got to the part that removes osxfuse and macfuse....

  2. git clone https://github.com/jollyjinx/encfs.macosx and run install.sh which downloads and runs the osxfuse installer and then installs encfs configured for osxfuse.

  3. Open osxfuse's pane in System Preferences and check the version.

  4. Reinstall Bitcasa.

  5. Check osxfuse's version again. Bitcasa had installed an older version on top of the one just installed, so I updated osxfuse back to the latest version.

After these steps, both Bitcasa and encfs worked for me. Most of the work involved here was done by Patrick Stein in creating the brew formula for encfs that uses osxfuse. If you want to follow the steps above, be sure to read over `install.sh` and `encfsmacosxfuse.rb` and make sure they look okay. I don't know how often the formula will be updated over time to keep it functional. As noted in the encfs.macosx usage notes, encfs drives can be mounted in OS X with, e.g.:

encfs ~/Dropbox/Documents.encfs ~/Dropbox/Documents.secure -- -o volname="Documents"

By default, encfs stores the encryption key for an encrypted folder in the .encfs6.xml file in the folder. If you don't want to keep the key in this folder (e.g. you want to save the encrypted files on a shared drive where you wouldn't want people to have access to the key), you can store the key somewhere else and then run encfs with the ENCFS6_CONGFIG flag like so:

ENCFS6_CONFIG="/Users/username/.keys/.encfs6.xml" encfs ~/Dropbox/Documents.encfs ~/Dropbox/Documents.secure -- -o volname="Documents"

After installing osxfuse, brew doctor will complain about the extra files that were installed into /usr/local. Unfortunately, you just have to learn to live with this. I don't understand why brew doesn't at least support osxfuse as a bottle (meaning that it just runs the binary installer rather than building the files itself), but I am sure there are good reasons. Here is what I get for brew doctor after doing the above steps:

Warning: Unbrewed dylibs were found in /usr/local/lib. If you didn't put them there on purpose they could cause problems when building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
    /usr/local/lib/libmacfuse_i32.2.dylib
    /usr/local/lib/libmacfuse_i64.2.dylib
    /usr/local/lib/libosxfuse_i32.2.dylib
    /usr/local/lib/libosxfuse_i64.2.dylib
Warning: Unbrewed .la files were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .la files:
    /usr/local/lib/libosxfuse_i32.la
    /usr/local/lib/libosxfuse_i64.la
Warning: Unbrewed .pc files were found in /usr/local/lib/pkgconfig.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected .pc files:
    /usr/local/lib/pkgconfig/osxfuse.pc