YonderGit: Simplified Git Remote Repository Management

One of the great strengths of Git is the multiple and flexible ways of handling remote repositories. Just like Subversion, they can be "served" out of a location, but more generally, if you can reach it from your computer through any number of ways (ssh, etc.), you can git it.

YonderGit wraps up a number of a common operations with remote repositories: creating, initializing, adding to (associating with) the local repository, removing, etc.

You can clone your own copy of the YonderGit code repository using:

git clone git://github.com/jeetsukumaran/YonderGit.git

Or you can download an archive directly here: http://github.com/jeetsukumaran/YonderGit/archives/master.

After downloading, enter "sudo python setup.py" in the YonderGit directory to install. This will just copy the "ygit.py" script to your system path. After that, enter "ygit.py commands?" for a summary of possible commands, or "ygit.py --help" for help on options.

Quick Summary of Commands

$ ygit.py setup  

Create directory specified by "REPO-URL", using either the "ssh" or local filesystem transport protocol, initialize it as repository by running "git init", and add it as a remote called "NAME" of the local git repository. Will fail if directory already exists.

$ ygit.py create 

Create directory specified by "REPO-URL", using either the "ssh" or local filesystem transport protocol, and then initialize it as repository by running "git init". Will fail if directory already exists.

$ ygit.py init 

Initialize remote directory "REPO-URL" as a repository by running "git init" in the directory. Will fail if directory does not already exist.

$ ygit.py add  

Add "REPO-URL" as a new remote called "NAME" of the local git repository.

$ ygit.py delete 

Recursively remove the directory "REPO-URL" and all subdirectories and files.

Valid Repository URL Syntax

Secure Shell Transport Protocol

  • ssh://user@host.xz:port/path/to/repo.git/
  • ssh://user@host.xz/path/to/repo.git/
  • ssh://host.xz:port/path/to/repo.git/
  • ssh://host.xz/path/to/repo.git/
  • ssh://user@host.xz/path/to/repo.git/
  • ssh://host.xz/path/to/repo.git/
  • ssh://user@host.xz/~user/path/to/repo.git/
  • ssh://host.xz/~user/path/to/repo.git/
  • ssh://user@host.xz/~/path/to/repo.git
  • ssh://host.xz/~/path/to/repo.git
  • user@host.xz:/path/to/repo.git/
  • host.xz:/path/to/repo.git/
  • user@host.xz:~user/path/to/repo.git/
  • host.xz:~user/path/to/repo.git/
  • user@host.xz:path/to/repo.git
  • host.xz:path/to/repo.git
  • rsync://host.xz/path/to/repo.git/

Git Transport Protocol

  • git://host.xz/path/to/repo.git/
  • git://host.xz/~user/path/to/repo.git/

HTTP/S Transport Protocol

  • http://host.xz/path/to/repo.git/
  • https://host.xz/path/to/repo.git/

Local (Filesystem) Transport Protocol

  • /path/to/repo.git/
  • path/to/repo.git/
  • ~/path/to/repo.git
  • file:///path/to/repo.git/
  • file://~/path/to/repo.git/

Setting Up Git to Use Your Diff Viewer or Editor of Choice

Git offers two ways of viewing differences between commits, or between commits and your working tree: diff and difftool.
The first of these, by default, dumps the results to the standard output.
This mode of presentation is great for quick summaries of small sets of changes, but is a little cumbersome if there are a large number of changes between the two commits being compared and/or you want to closely examine the changes, browsing back-and-forth between different files/lines, search for specific text, fold away or hide non-changed lines etc.
In these cases, you would like to use an external or third-party diff program/viewer to review the differences, and
Git offers two ways to allow for this.

The Less-Than-Ideal Approach

You can set a configuration variable to send the results to a third-party diff program by adding the following line to your “~/.gitconfig“:

    external = 

where “” is a script that invokes your diff program/viewer of choice.
The reason you need to use a wrapper script rather than the external program directly is because Git calls the program specified by passing it seven arguments in the following order: “path“, “old-file“, “old-hex“, “old-mode“, “new-file“, “new-hex“, “new-mode“.
So, depending on your diff program, you would need to filter out unneeded/unused arguments, or add switches/flags as appropriate.
For example, if you want to use Vim, the wrapper script may look something like:

#! /bin/sh
vimdiff "$2" "$5"

This approach is less than ideal, however, at least for me, because once you have configured your Git this way, then all invocations of “git diff” will launch the external applications.
And the fact is that there are many times (the majority, in my case) where this is simply overkill and the short summary in standard output does just fine.
You can, of course, still get the native Git standard output diff dump even with the external diff program configured as above by passing in an appropriate flag, but, trust me, this is a bit of a pain.

The Ideal Approach

Git, fortunately, offers a second approach: difftool.
This is essentially a wrapper around diff, taking the same arguments and options, but instead calls the external diff program/viewer by default.
This approach thus allows you to retain “git diff” for standard output reviews of changes, while unleashing the power of a more sophisticated diff program/viewer for more extended/flexible/complex reviews by invoking “git difftool“.

Git offers a range of difftools pre-configured “out-of-the-box” (kdiff3, kompare, tkdiff, meld, xxdiff, emerge, vimdiff, gvimdiff, ecmerge, diffuse, opendiff, p4merge and araxis), and also allows you to specify your own.
To use one of the pre-configured difftools (for example, “vimdiff”), you add the following lines to your “~/.gitconfig“:

    tool = vimdiff

Specifying your own difftool, on the other hand, takes a little bit more work, but is still pretty simple … IF you know how to do it.

I did not. And, unfortunately, the documentation did not help me very much.
It took quite a bit of Googling and experimentation before I figured it out.

You basically need to add the following lines to your “~/.gitconfig“:

    tool = default-difftool

[difftool "default-difftool"]
    cmd = default-difftool.sh $LOCAL $REMOTE

You can, of course, replace “default-difftool” with anything you care to name your preferred difftool, and “default-difftool.sh” with whatever you end up calling your wrapper script.

My difftool of choice is Vim, and, while “vimdiff” is a pre-configured option, I did not want to use it because I wanted the flexiblity to invoke MacVim when I am using my laptop but console Vim when I am working remotely on a Linux box (my Git configuration, and for that matter, my entire work environment from the shell to Vim to what-have-you, all 37MB, is shared across multiple machines … and all managed/synced using Git, of course).
So my wrapper script looks like the following:

#! /bin/bash

if [[ -f /Applications/MacVim.app/Contents/MacOS/Vim ]]
    # bypass mvim for speed
    VIMPATH='/Applications/MacVim.app/Contents/MacOS/Vim -g -dO -f'
elif [[ -f /usr/local/bin/mvim ]]
    # fall back to mvim
    VIMPATH='mvim -d -f'
    # fall back to original vim


And that’s all there is to it!

One Last Tweak

I find it very annoying to have to hit “” before moving on to the next file. The following lines added to your “~/.gitconfig” put a stop to that:

    prompt = false

Unconditionally Accepting All Merging-In Changes During a Git Merge

Merge conflicts suck. It is not uncommon, however, that you often just know that you really just want to accept all the changes from the branch that you are merging in. Which makes things a lot simpler conceptually. The Git documentation suggests that this can also be procedurally simple as well, as it mentions the “-s theirs” merge strategy which does just that, i.e., unconditionally accept everything from the branch that you are merging in:

$ git merge -s theirs 

Unfortunately, however, running the above command results in an error message along the line of “theirs” is not a known strategy. This is because, as discussed here (original reference here), this option has been removed from Git. I am sure the reasons for this are sound. But it is too bad that the documentation (as of my current installation, 1.7.7) has not been updated to reflect these changes. Really too bad. Really, really, really, really too bad. Because it takes a potentially dangerous, always stressful, and sometimes frustrating experience and makes it all the worse due to outright false documentation. Still, as an open source developer myself, I recognize that the time, energy, and effort demands of maintaining open source software have to be fitted in in between the demands of the other aspects of life, and how there consequently is sometimes considerable lag in updating what usually receives the lowest priority: the documentation.

In any case, luckily the page referred to previously provides some nice solutions for achieving the same effect as “-s theirs, which I am summarizing here for my own reference.

Method #1

git merge -s ours ref-to-be-merged
git diff --binary ref-to-be-merged | git apply -R --index
git commit -F .git/COMMIT_EDITMSG --amend

Method #2

git checkout MINE
git merge --no-commit -s ours HERS
git rm -rf .
git checkout HERS -- .
git checkout MINE -- debian # or whatever, as appropriate

Method #3

# get the contents of another branch
git read-tree -u --reset 
# selectivly merge subdirectories
# e.g superseed upstream source with that from another branch
git merge -s ours --no-commit other_upstream
git read-tree --reset -u other_upstream     # or use --prefix=foo/
git checkout HEAD -- debian/
git checkout HEAD -- .gitignore
git commit -m 'superseed upstream source' -a

Of these, I tried the first method, and it worked like a charm. YMMV.

List All Changes from a Git Pull, Merge, or Fast-Forward

When you pull and update your local, it would be nice to easily see all the commits that you have applied in the pull. Sure you can figure it by scanning through the git log carefully, but adding the following to your ‘~/.gitconfig’ gives you an easy way to see it in a glance:

    whatsnewlog = !"sh -c \"git log  --graph --pretty=format:'%Creset%C(red bold)[%ad] %C(blue bold)%h%C(magenta bold)%d %Creset%s %C(green bold)(%an)%Creset'  --abbrev-commit --date=short $(git symbolic-ref HEAD 2> /dev/null | cut -b 12-)@{1}..$(git symbolic-ref HEAD 2> /dev/null | cut -b 12-)\""
    whatsnew = !"sh -c \"git diff  $(git symbolic-ref HEAD 2> /dev/null | cut -b 12-)@{1}..$(git symbolic-ref HEAD 2> /dev/null | cut -b 12-)\""

When you pull, the HEAD of the current branch fast-forwards to the end of all the new commits. HEAD@{1} refers to the previous position of HEAD, so ‘git diff HEAD@{1}..HEAD shows you all the stuff in the current HEAD that was not in the previous HEAD position. The same applies to branch references, and the messy stuff ‘$(…)’ is simply to get the current branch name (using the branch name is better than HEAD, because it will have correct behavior even if you do a pull and checkout another branch).

The fancy colors (‘%C(…)’) will only work with Git 1.6.6 or later, so you can remove it if you do not want it.

Grepping in Git: How to Search Git Repository Revisions, Working Trees, Commit Messages, etc.

  • To search content of all tracked files in the current working tree for a pattern:

    git grep 
  • To search content of all commit messages for a pattern (‘-E’ for extended grep):

    git log [-E] --grep 
  • To search content of all commit diffs for lines that add or remove a pattern (‘-w’ for pattern only at word boundary):

    git [-w] log -G
  • To search content of entire working trees of previous revisions for a pattern:

    git grep  $(git rev-list --all)

Note that Git supports POSIX Basic Regular Expression. Which is pretty, well, basic. Not much more beyond matching beginning/ending of lines (‘^‘ and ‘$‘), the “match anything” dot (‘.‘), zero or more of preceding start (‘*‘), and some quantifiers. Took me a long time to figure out why a lot of my searches using some of the “fancier” operators (from a pre-1970 perspective) were not working.

`gcd` – A Git-aware `cd` Relative to the Repository Root with Auto-Completion

The following will enable you to have a Git-aware "cd" command with directory path expansion/auto-completion relative to the repository root. You will have to source it into your "~/.bashrc" file, after which invoking "gcd" from the shell will allow you specify directory paths relative to the root of your Git repository no matter where you are within the working tree.

gcd() {
    if [[ $(which git 2> /dev/null) ]]
        STATUS=$(git status 2>/dev/null)
        if [[ -z $STATUS ]]
        TARGET="./$(git rev-parse --show-cdup)$1"
        #echo $TARGET
        cd $TARGET
    if [[ $(which git 2> /dev/null) ]]
        STATUS=$(git status 2>/dev/null)
        if [[ -z $STATUS ]]
        TARGET="./$(git rev-parse --show-cdup)"
        if [[ -d $TARGET ]]
        dirnames=$(cd $TARGET; compgen -o dirnames $2)
        opts=$(for i in $dirnames; do  if [[ $i != ".git" ]]; then echo $i/; fi; done)
        if [[ ${cur} == * ]] ; then
            COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
            return 0
complete -o nospace -F _gcd gcd

Enhanced Git Log View Showing Symbolic References Associated With Each Commit


With multiple upstream repositories and branches, and different branches on different upstreams, an enhanced "log" view will help greatly in taking stock of everything. Adding the following line to your "~/.gitconfig" will give you a new command, "git slog" (for "short log") that does just that:



    # colorful 1-line log summary
    slog = log --pretty=format:'%Creset%C(red bold)[%ad] %C(blue bold)%h %Creset%C(magenta bold)%d %Creset%s %C(green bold)(%an)%Creset' --abbrev-commit --date=short

This command will provide a colorful one-line summary of the project’s commit history, showing not only the commit date, SHA-1, commit message and author, but, most importantly in this context, an indication of the symbolic references associated with various commits in nice, bright magenta.

So, for example, in this log we can see:

  • "(HEAD, master)", telling us that our current local HEAD and the tip of the local "master" branch are both on commit d685fb3.
  • "(origin/master, origin/HEAD)", telling us that the HEAD of the remote origin is two commits behind us, as is the tip of its "master" branch.
  • "(fasta-tweak)", telling us that we have a local branch, "fasta-tweak", with its tip at revision 1f87947.
  • "(dev/master)", telling us that the tip of the "master" branch is way behind everything else, on revision f070902a.
  • Various "(tag: ###)" entries, showing us the commits pointed to by various tags.

With a log summary such as this, managing multiple branches across multiple repositories is greatly facilitated.

Easily Create Clean Compressed Tarballs of Your Git Repository

Ideally, you could refer the whole world — or at least, the significant portion thereof that want your code — to your (public mirror) Git repository. But unfortunately, the whole world does not (yet) use Git ("I know it was you Fredo, I know it was you, and it breaks my heart."). Sad. Sooooo sad. But true. So the only recourse is for you to send these tortured souls an archived snapshot of your code via e-mail. I’ll pause now to let you finish retching/sobbing/lamenting/venting. … Back? Anyway, Git has a neat "archive" command that helps you create the required archive, but perhaps it does not have the most friendliest interface in the world. I’ve written some scripts to wrap the Git command to facilitate its use, which I share here.

Update 2011-07-07: I have re-written the original three scripts previously described here as a single script, incorporating some of the excellent suggestions that kind folks provided in the comments. Immediately below this is the new, unified, all-bells-and-whistles-included script. Simply name it something like “git-makedist”, switch on its executable bit, and place it somewhere on your system path. Reasonable names, filepaths, prefixes, etc. are all provided if you do not explicitly specify them. All of which means that, in most cases, once installed, simply invoking the command “git makedist” without any options or arguments from within a Git repository “just works ™”, resulting in a sensibly-named compressed archive of the current HEAD ready for distribution. At the same time, various flags and options allow you to fine-tune the operations, e.g. determine the compression/archive method, the path prefixes, the archive name, etc.

Update 2011-07-22, Excluding Files:
Sometimes, not all files need to make it out to an archive. You can use the “.gitattributes” file to control which files get archived. Simply add the paths/names/globs of the files or directories you would like to exclude, and add a “export-ignore” after it. For example:

.gitattributes export-ignore
.gitignore export-ignore
*.tmp export-ignore

Note that the “.gitattributes” file has to be committed for it to have an effect.

#! /bin/bash

compose_name() {
    if [[ $(which git 2> /dev/null) ]]
        STATUS=$(git status 2>/dev/null)
        if [[ -z $STATUS ]]
        TARGET="./$(git rev-parse --show-cdup)$1"
        cd $TARGET
        echo "$(basename $(pwd))"
        cd - > /dev/null

function usage {
    echo "`basename $0` [OPTIONS]"
    echo "Archives repository tree."
    echo " "
    echo "Options:"
    echo "  -f|--format FORMAT  ... Archive format ([targz], tarbz, zip)"
    echo "  -r|--revision REV   ... Revision (default=HEAD)"
    echo "  -p|--prefix PREFIX  ... prefix to add before directory paths in archive"
    echo "                          (default = repository name)"
    echo "  --no-prefix         ... suppress prefix before directory paths in archive"
    echo "  -n|--name NAME      ... Archive file basename"
    echo "                          (default = repository name)"
    echo "  -o|--output DIR     ... Archive output directory (default = current)"
    echo "  -h|--help           ... Show help"
    exit 1

while [ -n $1 ]; do
    case $1 in
        shift 2
        shift 2
        shift 2
        shift 2
        shift 2

if [[ -z $FORMAT ]]

if [[ -z "$REVISION" ]]

if [[ -z $REPONAME ]]
    if [[ -z $REPONAME ]]
        echo "FATAL: not a Git repository, or could not parse repository name (supply using '-p'/'--prefix' flag)"
        exit 1

if [[ -z "$OUTPUTDIR" ]]

if [[ $FORMAT == "zip" ]]
    if [[ $FORMAT == "tarbz" ]]
    elif [[ $FORMAT == "targz" ]]
        echo "FATAL: Unrecognized format '$FORMAT'"
        exit 1

if [[ $NOPREFIX == 1 ]]
elif [[ -z "$PREFIX" ]]

ARCHIVEBASENAME="$REPONAME-$(git describe --always)"
if [[ $FORMAT == "zip" ]]