I am always tweaking and tricking my bash environment. I hit the same issues again and again and I always have to look up the solution, time after time. This happens until I get annoyed enough to sit down – okay, generally I am already sitting down but you get the point – and create a custom function, put it in my .bashrc and deploy it to any machine I log on to.
In the hope my struggle for ultimate terminal efficiency is of help to other fellow command liners have a look at some tips and functions I use frequently. It would be cool if this would turn into a two-way dialog and you also suggest your own bash shortcuts at @durdn, @atlassiandev or in the comments below!
Without further ado, here’s what I’ve got today.
1. Add a line at the top of a file
I have to look up this every time. Here’s now to add a line at the top of a text file using sed:
sed -i '1s/^/line to insertn/' path/to/file/you/want/to/change.txt
2. Append text to a configuration file
Easy and well known, here’s how to append (>>) several lines to a file. It uses the “here document” syntax which allows you to embed a document into your source by specifying which word delimits the end of the file. Most commonly used word is EOF (aka “End Of File”):
cat >> path/to/file/to/append-to.txt << "EOF"
export PATH=$HOME/jdk1.8.0_31/bin:$PATH
export JAVA_HOME=$HOME/jdk1.8.0_31/
EOF
Everything in between the first EOF and the last gets appended to the file.
3. Recursive global search and replace
If you use Eclipse, IntelliJ or any other IDE you probably have powerful refactoring capabilities at your fingertips. But I bet sometimes you work on a tech stack that does not come with so advanced capabilities.
How do I do a global search and replace on a directory tree on the command line again? Please, don’t make me use Perl, what about find and sed? Thank you Stack Overflow:
# OSX version
find . -type f -name '*.txt' -exec sed -i '' s/this/that/g {} +
After a few times I have added a custom function to my .bashrc like the following:
function sr {
find . -type f -exec sed -i '' s/$1/$2/g {} +
}
And you use it like this:
sr wrong_word correct_word
4. Open a scratch file in vim and Dropbox
I used to loved the scratch facility of Emacs and I missed the ability to quickly create a temp file to type persistent notes on my vim setup. So I came up with these two quick bash functions which use openssl to generate a random alphabetic string as filename:
function sc {
gvim ~/Dropbox/$(openssl rand -base64 10 | tr -dc 'a-zA-Z').txt
}
function scratch {
gvim ~/Dropbox/$(openssl rand -base64 10 | tr -dc 'a-zA-Z').txt
}
By typing sc or scratch on my terminal a new gvim or macvim windows pops up with random temporary file stored in my Dropbox folder ready for my “insightful” notes (aka random trash).
5. Download a file following redirects, with https, security concerns
Output a page to stdout (updated!) following redirects and ignoring security exceptions:
curl -Lks <some-url>
Download a file following redirects and ignoring security exceptions:
curl -OLks <some-url/to/a/file.tar.gz>
I know, I know… there’s no need to note those flag down! It’s enough to read the very easy and short curl documentation! (humor engine engaged at full power).
6. Bashmarks for the win
If you don’t have bashmarks in your .bashrc yet, what are you waiting for? They are awesome. They allow you to save and jump back to directories you use often. I have a minimal setup for them in my configuration like the following, but the link above has a more robust solution you can load up in your .bashrc:
# USAGE:
# s bookmarkname - saves the curr dir as bookmarkname
# g bookmarkname - jumps to the that bookmark
# g b[TAB] - tab completion is available
# l - list all bookmarks
# save current directory to bookmarks
touch ~/.sdirs
function s {
cat ~/.sdirs | grep -v "export DIR_$1=" > ~/.sdirs1
mv ~/.sdirs1 ~/.sdirs
echo "export DIR_$1=$PWD" >> ~/.sdirs
}
# jump to bookmark
function g {
source ~/.sdirs
cd $(eval $(echo echo $(echo $DIR_$1)))
}
# list bookmarks with dirnam
function l {
source ~/.sdirs
env | grep "^DIR_" | cut -c5- | grep "^.*="
}
# list bookmarks without dirname
function _l {
source ~/.sdirs
env | grep "^DIR_" | cut -c5- | grep "^.*=" | cut -f1 -d "="
}
# completion command for g
function _gcomp {
local curw
COMPREPLY=()
curw=${COMP_WORDS[COMP_CWORD]}
COMPREPLY=($(compgen -W '`_l`' -- $curw))
return 0
}
# bind completion command for g to _gcomp
complete -F _gcomp g
7. Extract a column from a tabular output (my most used awk trick)
I use this daily. Really. I get some output and I only want the second column of it, or the third or whichever and typing out the full command is wordy:
#Sample output of git status -s command:
$ git status -s
M .bashrc
?? .vim/bundle/extempore/
# Remove status code from git status and just get the file names
$ git status -s | awk '{print $2}'
.bashrc
.vim/bundle/extempore/
Why not create a simple function that you can use anytime?
function col {
awk -v col=$1 '{print $col}'
}
This makes slicing columns so easy, for example you want to remove the first column? Easy:
$ git status -s | col 2
.bashrc
.vim/bundle/extempore/
8. Skip first x words in line
I am a fan of xargs, I think it’s almost as good as sliced bread. But sometimes you need to massage the list you get back from it, maybe skipping a few values? For example when you want to remove stale docker images and the first line is a heading you don’t need:
function skip {
n=$(($1 + 1))
cut -d' ' -f$n-
}
Here’s how to use it:
- The full tabulated command
docker imagesgives back:
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
<none> <none> 65a9e3ef7171 3 weeks ago 1.592 GB
<none> <none> 7c01ca6c30f2 3 weeks ago 11.1 MB
<none> <none> 9518620e6a0e 3 weeks ago 7.426 MB
<none> <none> 430707ee7fe8 3 weeks ago 7.426 MB
boot2docker/boot2docker latest 1dbd7ebffe31 3 weeks ago 1.592 GB
spaceghost/tinycore-x86_64 5.4 f47686df00df 7 weeks ago 11.1 MB
durdn/bithub latest df1e39df8dbf 8 weeks ago 100.9 MB
<none> <none> c5e6cf38d985 8 weeks ago 100.9 MB
nginx latest e426f6ef897e 12 weeks ago 100.2 MB
zoobab/tinycore-x64 latest 8cdd417ec611 8 months ago 7.426 MB
scratch latest 511136ea3c5a 20 months ago 0 B
- With the previous newt
colfunction I can get all the image ids like this:
$ docker images | col 3
IMAGE
65a9e3ef7171
7c01ca6c30f2
9518620e6a0e
430707ee7fe8
1dbd7ebffe31
f47686df00df
df1e39df8dbf
c5e6cf38d985
e426f6ef897e
8cdd417ec611
511136ea3c5a
- Now we are close I can almost remove them all with:
docker images | col 3 | xargs
IMAGE 65a9e3ef7171 7c01ca6c30f2 9518620e6a0e 430707ee7fe8 1dbd7ebffe31 f47686df00df df1e39df8dbf c5e6cf38d985 e426f6ef897e 8cdd417ec611 511136ea3c5a
- But there is that spurious “IMAGE” which I’d like to, well,
skip:
docker images | col 3 | xargs | skip 1
65a9e3ef7171 7c01ca6c30f2 9518620e6a0e 430707ee7fe8 1dbd7ebffe31 f47686df00df df1e39df8dbf c5e6cf38d985 e426f6ef897e 8cdd417ec611 511136ea3c5a
- Now I can remove all of them by feeding the result to
docker rmi(docker remove image command):
docker rmi $(docker images | col 3 | xargs | skip 1)
9. Create your very own command package
in bash it’s incredibly easy to create your own command suite, names paced however you please. Have a look at some things I put in mine:
function dur {
case $1 in
clone|cl)
git clone git@bitbucket.org:nicolapaolucci/$2.git
;;
move|mv)
git remote add bitbucket git@bitbucket.org:nicolapaolucci/$(basename $(pwd)).git
git push --all bitbucket
;;
trackall|tr)
#track all remote branches of a project
for remote in $(git branch -r | grep -v master ); do git checkout --track $remote ; done
;;
key|k)
#track all remote branches of a project
ssh $2 'mkdir -p .ssh && cat >> .ssh/authorized_keys' < ~/.ssh/id_rsa.pub
;;
fun|f)
#list all custom bash functions defined
typeset -F | col 3 | grep -v _ | xargs | fold -sw 60
;;
def|d)
#show definition of function $1
typeset -f $2
;;
help|h|*)
echo "[dur]dn shell automation tools"
echo "commands available:"
echo " [cl]one, [mv|move]"
echo " [f]fun lists all bash functions defined in .bashrc"
echo " [def] <fun> lists definition of function defined in .bashrc"
echo " [k]ey <host> copies ssh key to target host"
echo " [tr]ackall], [h]elp"
;;
esac
}
With the above I can copy my ssh key to any site just by typing dur key user@host.
Conclusions
Peruse the custom functions in my .bashrc or come up with your own. Do you have other neat tricks or short functions that help you in your daily terminal hackings? Let me know in the comments below or at @durdn on twitter. I’m always on the look for new ideas.