Estimate Time for Job Completion (With Progress Updates) When Tar’ing Huge Directories

For the sake of future me, I am recording this here, the coolest shell trick I've learned this year: (Linux): tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '{print $1}') | gzip > big-files.tar.gz (OSX): tar cf - /folder-with-big-files -P | pv -s $(($(du -sk /folder-with-big-files | awk '{print $1}') * 1024)) | gzip > big-files.tar.gz with output looking like: 4.69GB 0:04:50 [16.3MB/s] [==========================> ] 78% ETA 0:01:21 Requires Read more [...]

The Traveler’s Restaurant Process — A Better Description of the Dirichlet Process for Partitioning Sets

[latexpage] I. "Have Any of These People Ever Been to a Chinese Restaurant?" The Dirichlet process is a stochastic process that can be used to partition a set of elements into a set of subsets. In biological modeling, it is commonly used to assign elements into groups, such as molecular sequence sites into distinct rate categories. Very often, an intuitive explanation as to how it works invokes the "Chinese Restaurant Process" analogy. I have always found this analogy very jarring and confusing, Read more [...]

“Joy Plots” — Great Plot Style for Visualizing Distributions on Discrete/Categorical or Multiple Continuous Variables

R doing what R does really, really, really, really, really, really, *R*eally well: visualization. Folks, this might be THE plot to use to visualize distributions of discrete/categorical variables or simultaneous distributions of multiple continuous variables, replacing or at least taking up a seat alongside the violin plots as the current best approach IMHO. Source code repository: ggjoy Example of use (EDIT: This plot style is named after the "Joy Division", due to a similar Read more [...]

“Pre-Columbian Mycobacterial Genomes Reveal Seals As A Source Of New World Human Tuberculosis”

When, in 1994, definitive evidence of tuberculosis in humans was reported from pre-Columbian America, it was a startling. Conventional understanding had pegged tuberculosis as part of the new, exotic, and (to immunologically-naive populaces) deadly menagerie of pathogens brought by Europeans over to the Americas. While there were suggestions of pre-Columbian tuberculosis in the Americans, these were based on lesions on bones, which were ambiguous. Unlike previous cases, however, the Chiribaya mummy Read more [...]

Multispecies Coalescent Species Delimitation: Conflating Populations with Species in the Grey Zone (Evolution 2017 Talk)

Folks! The always fantastic Evolution meetings were a blast. So many great talks, and, perhaps more importantly, great catching up with so many friends, collaborators, and colleagues! I presented a talk on our PNAS paper showing how the Multispecies Coalescent model, when used for "species" delimitation, actually delimits Wright-Fisher populations. Titled "Multispecies Coalescent Species Delimitation: Conflating Populations with Species in the Grey Zone", the entire talk can Read more [...]

“Phylogenomics reveals rapid, simultaneous diversification of three major clades of Gondwanan frogs at the Cretaceous–Paleogene boundary”

Some nice work that ties the timing of the radiation of three independent lineages of frogs, constituting the majority of modern living frogs, to about the time the major groups of dinosaurs took a hit (literally and figuratively!). Compelling and interesting story, with lots of intriguing follow-up questions. A more general article covering the findings is available here. Yan-Jie Feng, David C. Blackburn, Dan Liang, David M. Hillis, David B. Wake, David C. Cannatella, and Peng Zhang. Read more [...]

Solving the “Could not find all biber source files” Error

Biblatex is a fantastic bibliography/citation manager for LaTeX. It trumps the older bibtex for its much easier customizability and configuration. It does however, have one bug that can be very perplexing to figure out due to the misleading error message that results: "Could not find all biber source files". At first glance this message seemed straightforward enough to send me poking about the project file structure and build system, checking paths and names. When all that seemed intact, I started Read more [...]

Invisible Photographer Meets Vanishing Fox

My absolute favorite wildlife encounter in the Pantanal was with a central South American endemic Crab-Eating Fox (Cerdocyon thous). I had left the group behind in the safari vehicle on the road to try and close in on some jabiru in the marsh on foot. I was creeping about, camera + lens in hand, when suddenly, I saw her staring at me through the marsh vegetation. I froze. After a few moments, I realized that it seemed that while she sensed something, she could not actually see me: she kept sniffing Read more [...]

Is that an archosaur in your Pantanal, or are you just happy to see me?

Finally getting around to finishing processing some of my Pantanal wetland photographs, and realized that despite being famous for its endemic birds, I was drawn more to the other clade of archosaurs in the region, the Crocodylia, represented by the Black Caiman, Caiman yacare. The Pantanal population of caimans is the largest single crocodiilan population on the planet. Read more [...]

More Sandhill Cranes!

Love these birds! More Sandhill Cranes from the kayak. This time I deliberately avoiding going too close to where I last saw the nesting pair, and instead hit a different area of the marsh. Here I stumbled upon a colony of 8-10 individuals. (The one on the right seems to be complaining about its day to the others, who are all dutifully listening, The middle one has completely zoned out, though, and is daydreaming about cornfields while waiting for the rant to end.) Read more [...]

Sandhill Cranes Parents and Chick

AMAZING experience! So, I was using my kayak to stalk a Great Blue Heron in the marshes at the other end of the lake behind our house and the guy kept drifting deeper and deeper into the marsh, till I could not follow any more (less than a 10th of an inch water, and chock full of vegetation; next time, I am getting a push pole!). So I turn around to head back, when I see these guys almost right next to me! A pair of Sandhill Cranes. AWESOME!!! I spent a while photographing them, watching them alternative Read more [...]

Evolution of Bioluminescence in Millipedes

Walk deep into a rainforest at night. Switch off your headlamps. And wait with open eyes. At first, it is so pitch black that you cannot see your own hand if you wave it in front of nose (as Bilbo might have said). As your eyes get accustomed to the darkness, you will realize one thing. Everything glows. Everything. There is fine fuzzy layer of bioluminescent fungus covering dead leaves and the bark of trees, so you can almost make out the forest like some one has traced it out in ghostly yellow-green Read more [...]

Vim: Insert Mode is Like the Passing Lane

Insert mode is not the mode for editing text. It is a mode for editing text, because both normal and insert modes are modes for editing text. Insert mode, however, is the mode for inserting new/raw text directly from the keyboard (as opposed to, e.g., from a register or a file). Thus, you will only be in insert mode when you are actually typing in inserting (raw) text directly. For almost every other editing operation, normal mode is where you will be. Once you grok this you will realize that, Read more [...]

From Acolyte to Adept: The Next Step After NOP-ing Arrow Keys in Vim

We all know about no-op'ing arrow keys in Vim to get us to break the habit of relying on them for inefficient movement. But, as this post points out, it is not the location of the arrow keys that makes them inefficient, but the modality of the movement: single steps in insert mode is a horrible way to move around when normal mode provides so much better functionality. But here is the thing: while normal mode provides for much better and more efficient ways to move around than insert mode, Read more [...]

Laika: A Sad, Unnecessary Death

I think our space programme is one of our species' greatest achievements. It does have a sordid past, though, with roots in war, conflict, aggression, violence, paranoia, and narrow-minded parochial/tribal brutality. Which makes it all the more remarkable that I think it unites us as a species now, when it was born of such acrimonious and savage division. But probably one of the greatest crimes of the early days our species' space programmes is the death of Laika. A congenial, friendly, Read more [...]

Comet Landing Pride: Rosetta Mission to Churyumov-Gerasimenko

I am always resistant to imparting any exceptionalism to our species, considering it some sort of narrow-minded parochialism based on the accident of the restricted perspective we have when we try to place ourselves in context with the rest of nature. But every time I look at at the achievements of our space programmes, I find it hard not take pride in our species' achievements in this domain (whatever else the horrible things we do): this is truly where our reach exceeds our grasp, but again and Read more [...]