Estimate Time for Job Completion (With Progress Updates) When Tar'ing Huge Directories

For the sake of future me, I am recording this here, the coolest shell trick I’ve learned this year: (Linux): tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '') | gzip > big-files.tar.gz (OSX): tar cf - /folder-with-big-files -P | pv -s $(($(du -sk /folder-with-big-files | awk '') * 1024)) | gzip > big-files.tar.gz with output looking like: 4.69GB 0:04:50 [16.3MB/s] [==========================> ] 78% ETA 0:01:21 Requires ‘pv’: https://github.

Read more

'Joy Plots' -- Great Plot Style for Visualizing Distributions on Discrete/Categorical or Multiple Continuous Variables

R doing what R does really, really, really, really, really, really, *R*eally well: visualization. Folks, this might be THE plot to use to visualize distributions of discrete/categorical variables or simultaneous distributions of multiple continuous variables, replacing or at least taking up a seat alongside the violin plots as the current best approach IMHO. Source code repository: ggjoy Example of use (EDIT: This plot style is named after the “Joy Division”, due to a similar graphic on one of their album covers.

Read more

Solving the 'Could not find all biber source files' Error

Biblatex is a fantastic bibliography/citation manager for LaTeX. It trumps the older bibtex for its much easier customizability and configuration. It does however, have one bug that can be very perplexing to figure out due to the misleading error message that results: “Could not find all biber source files”. At first glance this message seemed straightforward enough to send me poking about the project file structure and build system, checking paths and names.

Read more

Building GCC From Scratch Natively on OSX 10.11 (El Capitan) and Above

With every iteration of their desktop operating system, Apple seems more and more determined to try new and novel ways to irritate me. The rootless security model that prevents anyone from writing to ‘/usr‘ (except for ‘/usr/local’; though there is no way for you to re-create this directory if you wipe it). The big problem is that the build process of GCC requires that ‘/usr/include’ exists, and the OSX 10.11 security model does not allow you to create it.

Read more

Vim: Insert Mode is Like the Passing Lane

Insert mode is not the mode for editing text. It is a mode for editing text, because both normal and insert modes are modes for editing text. Insert mode, however, is the mode for inserting new/raw text directly from the keyboard (as opposed to, e.g., from a register or a file). Thus, you will only be in insert mode when you are actually typing in inserting (raw) text directly. For almost every other editing operation, normal mode is where you will be.

Read more

From Acolyte to Adept: The Next Step After NOP-ing Arrow Keys in Vim

We all know about no-op’ing arrow keys in Vim to get us to break the habit of relying on them for inefficient movement. But, as this post points out, it is not the location of the arrow keys that makes them inefficient, but the modality of the movement: single steps in insert mode is a horrible way to move around when normal mode provides so much better functionality.

Read more

Setting up a Python Scientific Environment (NumPy, SciPy, pandas, StatsModels, etc.) in OS X 10.9 Mavericks

It is better than a nightmare from which you cannot wake up … Install Homebrew: $ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)" Find problems and fix them (typically resulting from Homebrew becoming very stroppy if it does not have exclusive access to “/usr/local”): $ brew doctor VERY IMPORTANT NOTE: Make ABSOLUTELY sure that the DYLD_LIBRARY_PATH environmental variable is NOT set. Having this set will cause all sorts of problems as conflicts arise between libraries that homebrew installs that some of your other packages need (e.

Read more

Taking it to a 11: Dramatically Speeding Up Keyboard/Typing Responsiveness in OSX

If you use a Mac/OSX, then enter the following commands in your shell and reboot: $ defaults write -g KeyRepeat -int 0 $ defaults write -g InitialKeyRepeat -int 15 If you live in a text editor or the shell, or otherwise spend most of your typing hammering away at the keyboard like I do, then this makes an absolutely wonderful difference in the responsiveness of any typing activity. It will make your previous typing feel like you were pecking away in slow motion at the bottom of a pit of cold tar!

Read more