A weirdly named site with lots of boring content!

Useful Ubuntu Linux Commands

Linux commands can be far from intuitive, with files related to a single application seemingly scattered throughout various locations, such as usr, bin, etc, var. This can make it difficult to recall how to invoke certain essential but not every-day commands. This page lists these commands in a single place.
compton, 6 September 07
Updated 17 December 15

Skip Certain Directories in find

Sometimes when running find, you don't want to look in certain directories e.g. Composer's vendor dir, or Subversion's .svn dir. Here's how you can do that:
find /sourcecode/ \( -path '*vendor*' -o -path '*.git*' -o -path '*.svn*' \) -prune -o -name '*.php' -exec grep -H MyHiddenClass {} \;

Copy All Modified Files in an SVN Repo over FTP

This method uses cURL to do the transfer, as it allows us to do it in a single line. We also use the -I argument for xargs to specify a substitution string for the input strings. Run this from the root of your local repo:
svn st | awk '{ print $2 }' | xargs -I {} curl -T {} ftp://username:password@ftp.host.com:/path/to/remote/repo/{}
Note that this will upload all files that appear in the SVN status output - not only new and modified files but also files with conflicts, so run svn st first to ensure only files you want to upload are listed.

How to Search within Results for Files Containing a Second Search String

The following command uses find combined with grep via the exec argument to search all PHP files for a search string, and then feeds the names of matching files to grep to search within them for a second string, using xargs:
find /path/to/search/ -type f -name "*.php" -exec grep -l 'Search String 1' {} \; | xargs grep -H 'Search String 2'

Fix PHP Files with Windows-1252 charset saved as UTF-8

During a large-scale customer site migration, some files which had been saved in Windows 1252 encoding were moved and saved as UTF-8 without conversion. This led to characters such as the apostrophe, em-dash and ellipsis appearing as black diamonds when served through Apache. iconv can be used to convert them into proper UTF-8 encoding like so:
find /srv/http -name "*.php" | while read file; do cp -f $file $file".old.php"; iconv -f WINDOWS-1252 -t UTF8 $file -o $file."utf8"; mv -f $file."utf8" $file; done
You'll need to change /srv/http above to the root of the site you want to convert. Also, because iconv doesn't have a 'convert in place' option, each converted file is saved with a .utf8 extension, and then moved on top of the original file. To be on the safe side, a backup of the original file is taken first, and saved with the extension .php.old.php.

Filter a list of domains that resolve to a given IP

Given a list of domain names in the file /tmp/doms, one per line, the following command will output only those which resolve to the IP specified:
for dom in `cat /tmp/doms`; do if [[ $(dig +short a $dom) = 192.168.0.100 ]]; then echo $dom; fi; done

Delete All Subversion Branches under a Specified Folder

This process can be automated by using svn rm in its second mode - deleting a URL - as this can be performed in a single step with a commit message specified on the command line. It's necessary to remove the trailing slash which svn ls appends using bash's split string operator otherwise svn rm won't recognise the argument as a URL:
svn ls ^/branches/experimental | while read line; do svn rm -m 'Bulk removing experimental branches' http://svn.server/repository/branches/experimental/${line%/*}; done;

Remove all SVN branches that have been merged to trunk

Buidling on the previous command, the following adds a mergeinfo check to only remove branches that have merges onto trunk:
svn ls ^/branches/myname | while read line; do mrevs=$(svn mergeinfo --show-revs=merged ^/branches/myname/${line} ^/trunk); if [ -n "$mrevs" ]; then echo $line' has merges'; svn rm -m 'Removing branches that have been merged to trunk' http://svn.server/repository/branches/myname/${line%/*}; fi; done;

Delete files Older than a Certain Age

This command will delete all files in the current directory that have not been modified in the last 30 days:
find . -maxdepth 1 -type f -not -name ".*" -mtime +30 -delete
Note the use of -not -name to exclude hidden files. If you want to test what it's going to splat before it does it (always a good idea), run the same command but using the ls action in place of delete:
find . -maxdepth 1 -type f -not -name ".*" -mtime +30 -ls

Pipe the Output of a text file into find


The problem with piping stuff into find is that the path to search is often not the last item on the command line. xargs and its replace option can help. With the replace option, you use the -i switch to specify a name that xargs then replaces with each line of the incoming input like so:
cat inputfile | xargs -ipath find -L path -type d

Find the Most Recently Modified File in a Directory Tree


First we use find to get all files which are not directories and output their UTC timestamp followed by their full path. This output is piped into sort using the -r switch to sort in reverse order, then passed into head with -1 to just give us the very first name produced:
find /var/log/ -not -type d -printf "%T+ %p\n" | sort -r | head -1
Use the -path switch along with -prune to tell find to ignore certain filepaths:
find /var/log/ \( -path */journal/* -o -path */samba/* \) -prune -a -not -type d -printf "%T+ %p\n" | sort -r | head -1
Note the use of the -o and -a logical operators to join expressions for find.

List the Targets of All Symlinks under a Particular Directory


Simply use find with the -exec option:
find /path/to/dir -type l -exec ls -l '{}' \;
Note the empty curly braces which denote where the filenames returned by find should be used, and also the final, escaped semicolon which denotes the end of the -exec command.

Remove File Associations created by WINE


Get rid of all the associations with the following:
rm -f $HOME/.local/share/applications/wine-extension-*
If you want to just remove one or two associations, list the directory contents, and the file names make it quite clear which file deals with which extension.

Recursively Copy Read-Only Files while Preserving File Attributes


Sometimes you might find yourself needing to copy a bunch of files organised in a directory tree from one place to another, keeping the files' positions in the tree intact along with their file attributes. If the files are not read-only, this is a simple matter of copying or moving the parent folder to the destination location. If the read-only bit is set, it's not so easy, and the following combination of find, rm and mv will do the job:
find . -type f | while read line; do rm -f /path/to/destination/$line; mv $line /path/to/destination/$line; done

How to Change X org Screen Resolution


Use xrandr for this. Run it without arguments to see what resolutions are supported by the attached monitor. Then run it again using the -s switch to specify the resolution you want:
xrandr -s 1920x1080

How to Test the First Character of a String on the CLI


The ${VARNAME::1} syntax will give you the first character in $VARNAME. For instance, say you want to list all files and directories that begin with a digit in the current directory:
for FILENAME in *; do if [[ ${FILENAME::1} =~ [[:digit:]] ]]; then ls -ld "$FILENAME"; fi; done
Notice we're using the =~ operator to do a RegEx match, against the POSIX [:digit:] character class. The double colon and curly brace notation ${VARNAME::1} to get the first character is in fact just a specific usage of the bash substring operator. For instance, if you want to find the first 3 characters of a variable, simply change the 1 to a 3 and use ${VARNAME::3}. This is in fact equivalent to ${VARNAME:0:3} meaning 'get the 3 characters of $VARNAME beginning at offset 0'. Thus, you can find the 4th and 5th characters of a variable with ${VARNAME:3:2} (NB character 4 is at offset 3 because of zero-based counting).

How to Split a String In bash


There are two operators that we can use to split a string on a given delimiter character, % and #. They are used in conjunction with the curly bracket operators to get the portion of a string before the delimiter (%), or the portion after (#). For instance, you could split filenames into the main part and the extension like this:
ls -1 | while read line; do echo 'filename bit: '${line%.*}; echo 'extension: '${line##*.}; done;
Note that the second example, which gets the extension, uses a double ##, which means 'split and return the shortest possible substring', while a single one would return the longest possible substring. Obviously it only makes a difference when there's more than one occurence of the 'split character'.

Set Page Up and Down keys to Search bash History


Create a file in your home directory called .inputrc containing the following lines:
"\e[5~": history-search-backward "\e[6~": history-search-forward
Now type the first few letters of a command, press Page Up and bash will show you the last command you entered that began with those letters. Press again for the previous match, and so on.

Increase the Sudo Timeout


After you run a terminal command as root by prepending it with sudo, you can execute further commands with sudo without reentering your password as long as each successive command is no more than 5 minutes apart, by default. You can change this default by adding a line like that below to your sudoers file:
Defaults:myusername timestamp_timeout=-1
By setting the timeout to -1 as above means that there is no timeout, and you only need enter your root password the first time you run sudo in any terminal session. Alternatively, use a positive integer to specify how many minutes you want the timeout to be. Don't forget that the right way to edit your sudoers file is by using visudo.

Recursively Applying chmod To Files Only


There are often occasions when you need to ensure all files in the current folder and all subfolders have specific access attributes, but chmod's -R flag would also apply the same attributes to directories. This is usually a problem because directories need the executable bit set if they are to be accessible. The solution is to use find in combination with chmod:
find .. -type f -print0 | xargs -0 chmod 755
Use -type d if you want to apply chmod only to directories rather than only to files.

Handy Bash Keyboard Shortcuts


Ctrl + LClears the Screen, similar to the clear command
Ctrl + ZPuts whatever you are running into a suspended background process. fg restores it.
Ctrl + WDelete the word before the cursor
Ctrl + UClears the line before the cursor position. If you are at the end of the line, clears the entire line.
Ctrl + KClear the line after the cursor
Ctrl + T Swap the last two characters before the cursor
Esc + TSwap the last two words before the cursor
Ctrl + _Undo
Alt + .Insert last word from history at the cursor

Clean Deadwood out of SVN Repo


It's not uncommon for extraneous files to find their way into a subversion repository over time. Backups, test scripts, IDE setting files and folders; all these and more can be found washed up in your once pristine repos. The more people you have in your team, the more likely this is to happen, and the more varied the deadwood flies will be. Often there will be multiple files with a similar filename scattered throughout, such as the subfolders called _notes that dreamweaver creates inside every folder.

We can splam all these in one go with a little CLI Fu. This bash command will remove all _notes folders from subversion, and then delete them from disk:
find . \( -name _notes! -name "*.svn*" \) | while read line; do svn delete "$line"; rm -rf "$line"; done
Obviously, you'll need to do a commit afterwards, and you'll probably want to do an update immediately beforehand.

Create a TAR archive with a Folder Tree starting at the Current Working Directory


Use the anchored option as below:
tar -cz --anchored -f archive.tar.gz folder1/subfolder1/* folder1/subfolder2/* folder2/* index.html style.css

Find files in a directory that are mentioned within files in another directory

ls srcdir | while read filename;do find targetdir -type f -name "*.php" -exec grep -o "$filename" {} \; ; done | uniq -c
This command lists all the files inside the directory called srcdir, and then searches all files in targetdir, including those inside subdirectories, looking for any that contain the filenames from srcdir. It outputs just the names of files from srcdir that are mentioned within other files, along with a count of how many times each was found.

This is useful when trying to determine if any files in a directory are no longer in use, ie no longer linked to from (or included by) other files.

Get File From an arbitrary SVN Repo

svn cat svn://myserver.dev/trunk/filefromtrunk.php > otherverson.php

Get SVN History Since a Specific Revision

svn log -r 31:HEAD
This will retrieve all commit messages, dates and users for all commits from revision 31 inclusive. You can substitute another revision number in place of HEAD if you want to limit to a range of revisions.

List All Files Modified Since a Specific SVN Revision


Basically use the same SVN command as above, but with the verbose switch and then feed into grep, sort and uniq to get a list of what files were modfiied, added or deleted:
svn log -r 31:HEAD -v | grep -E ' (M|A|D) /' | sort -k2 | uniq

Find SVN Revisions Committed by a Specific User


Do this by passing the SVN log through grep:
svn log svn://server/repo | grep -i insert_username_here
This will just show details such as the date, time and revision number from the commit message. You could use the -A2 switch on grep to view the first line of the commit message for each change, but this will take up screen space of course, and can easily eat up all your scrollback if this is not a new project and has a lot of history in the log. Alternatively you can just view the comment for a specific commit by replacing the XXX below with the revision number:
svn log -rXXX http://bacon.server.dev/svn/central
SVN also has the -lNN option which will limit the response to NN lines, but of course this is before it's further filtered by grep.

Find a server's external IP address via DNS from CLI

wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//'

Read a file and perform a command for each line


The following trivial example simply copies a file from inputfile to outputfile, but naturally you can replace the echo command with any other of your choosing.
cat inputfile | while read line;do echo $line >> outputfile; done

Useful Additions to .bashrc

# set xterm title and shell prompt export PS1="\[\e]2;\u@\H \w\a\e[31;1m\]\u@\H \w\[\e[0m\] "   # Use Colours in Man Pages export LESS_TERMCAP_mb=$'\E[01;31m' # begin blinking export LESS_TERMCAP_md=$'\E[01;34;5;74m' # begin bold export LESS_TERMCAP_me=$'\E[0m' # end mode export LESS_TERMCAP_se=$'\E[0m' # end standout-mode export LESS_TERMCAP_so=$'\E[38;5;227;48;5;124m' # begin standout-mode - info box export LESS_TERMCAP_ue=$'\E[0m' # end underline export LESS_TERMCAP_us=$'\E[04;36;5;146m' # begin underline   # ignore duplicates and any commands you enter where the first character is a space HISTCONTROL=erasedups:ignorespace   # ignore these commands in history file HISTIGNORE=l[sla]:cd:pwd:exit:clear   # preserve history across multiple terminals shopt -s histappend PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

Cool Stuff to put in ~/.inputrc

.inputrc holds config options for readline, the bash command line editor.
set expand-tilde on   # Cycle through autocomplete with TAB / Shift TAB "\e[Z": "\e-1\C-i" TAB: menu-complete   # Autocomplete from history with PgUp / PgDn "\e[5~": history-search-backward "\e[6~": history-search-forward   # mappings for Ctrl-left-arrow and Ctrl-right-arrow for word moving "\e[1;5C": forward-word "\e[1;5D": backward-word "\e[5C": forward-word "\e[5D": backward-word "\e\e[C": forward-word "\e\e[D": backward-word

Set Disk Checking to Once per Month


Ubuntu defaults to running a disk check every 30 boots, or once a month (I think). The following command will set it to just check once per month, regardless of how many times you reboot:
sudo tune2fs -c 0 -i 1m /dev/sda1

Search within Selected Files Recursing Through Folders


grep has the -r option to do a recursive search through folders under the specified path. However it won't work in conjunction with wildcards such as *.java. To do searches such as this, you need to use it in combination with find and xargs like so:
find ~/startfolder -type f -name '*.java' -print0 | xargs -0 grep 'searchpattern'

Create a tar.gz Compressed Archive

tar -cvzpf archive.tar.gz sourcedir1 sourcedir2
The -c option means create a new archive, -v is for verbose, -z indicates it should be a gzipped archive, -p means preserve permissions on files, and -f means use the specified archive file.

List contents of a tar.gz Compressed Archive

tar -tzf archive.tar.gz

Unzip a ZIP archive

unzip -d /path/to/destination /path/to/zip

List All directories containing certain files

find . -type f -name '*.py' |sed 's#\(.*\)/.*#\1#' |sort -u
This will list all directories under the current directory which contain python files.

Flush bash history


The bash shell keeps a record of commands that have been used in the file ~/.bash_history. However the commands used in the current session are not written to this file until you log out of the current session. If you wish to flush the history to this file before logging out, use the following:
history -a
'a' here stands for append - so this is the regular behaviour. If you want, you can use 'w' instead, which will replace the current bash history file with the session history.

Exit a bash Session without Saving History


If you don't want to leave certain sensitive commands in the bash history file, just exit your session with the following:
unset HISTFILE && exit
This leaves all the previous history intact. If you have accidentally placed some sensitive information in the history file, you can remove the sensitive info using sed like so:
sed -i s/SENSITIVESTRING//g ~.bash_history
The -i option stands for 'in place', and you need to specify it when you want sed to replace strings in a file, as opposed to, for instance, piping the altered file elsewhere. If you alter the contents of the bash history file as above, don't forget to exit your session using the preceding commands so you don't leave the sed command containing the sensitive string in history.

List the most recently modified file

ls -lt | grep ^[^dt] | head -1

Switch to a Virtual Terminal from the command line

sudo chvt 1

Display a Unix timestamp in a readable form

date -d @1193144433

Reset WINE (equivalent to a reboot in Windows)

wineboot

Restart the pulseaudio deamon

pulseaudio -k; pulseaudio -D

Restart Apache2 on Ubuntu

sudo /etc/init.d/apache2 restart

View Apache2 error log

gedit /var/log/apache2/error.log
This location is also where apache keeps its access.log file.

Run Disk Check on Next Reboot


Sometimes you might find that having a file named forcefsck in the root directory will make fsck run a check on the next boot:
sudo touch /forcefsck
For me though, this doesn't seem to work very reliably. In such cases, adding fsck.mode=force to the kernel line in Grub does work, and a check will be performed on every available partition.

Reload Apache2 Config Settings

sudo /etc/init.d/apache2 force-reload

Install Apache2 Module (eg ModRewrite)


sudo a2enmod module_name

eg sudo a2enmod rewrite

Then force Apache to reload config settings as above.

Regenerate the fonts cache:

sudo fc-cache -fv
The default location of system fonts is /usr/share/fonts/. Different font types are grouped into subfolders off here.

Create a symbolic link


ln -s /path/to/target /path/to/linkname
You can omit the final linkname argument if you want to create a link in the current directory with the same name as the target directory.

Change the target of an existing symbolic link


ln -sf /path/to/target /path/to/linkname
The 'f' option (or --force) actually overwrites any existing file with the same name & location as the newly created link. If not given, it should just hide any such files until the link is destroyed.

The effect is that if the link already exists, this command will change it so it links to the specified target location. If it doesn't already exist, it will be created as normal.

Copy a file over a network via SSH


ssh -l {login_name} 192.168.0.XXX cat /path/to/target > /path/to/outputfile

Shortcuts for Eclipse IDE


Lower Case: CTRL+SHIFT+Y
Upper Case: CTRL+SHIFT+X

Jump to Matching Bracket: CTRL+SHIFT+P

Copy current line(s) and place them above/below: CTRL+ALT+UP/DOWN
Move current line(s): ALT+UP/DOWN

Comment/uncomment current line(s): CTRL+/

Switch to next/prev tab: CTRL+PgUp/PgDn

Close current file: CTRL+W

Find open file: CTRL+E

Delete to end of line: CTRL+SHIFT+DELETE
Delete previous word: CTRL+BACKSPACE

Import Gzipped MySQL Database on the command line



gunzip < /path/to/database.sql.gz | mysql -u root -p -D databasename

The final -D databasename option for the mysql client selects the named database, and is only necessary when the gzipped SQL file doesn't begin with a USE databasename command.

Unix if conditional single-line syntax



The key to rememeber is that commands must be separated either by a newline or a semi-colon, and that if, then and fi are all commands.
if [ -d somedir ]; then ls somedir/; fi;

Split a File into Many Files each containing one line from the first


That heading pretty much describes it. Say for instance you have a list of hostnames in a text file called /tmp/hostlist, you can run it through the following command to generate a list of consecutively-numbered files each containing one hostname:
CNTR=1; cat /tmp/hostlist | while read -r line; do echo "$line" > /tmp/host$CNTR; CNTR=$((CNTR+1)); done

Unix for loop syntax



The following returns every line in all files called build.mk in all subdirectories of the current working directory which contain the text 'bird', along with the name of each file:
for filename in `ls *.mk`; do grep -H bird $filename; done
Typical output might be:

./src/mozilla/extensions/build.mk:# Ben Turner mozilla@bongbirdnest.com

The name and path of each file appears, a colon then the actual line containing the string 'bird'.

Mount an ISO Image in Unix



sudo mount -o loop /path/to/iso /path/to/mount/point

Replace Underscores with Spaces in File and Directory Names

ls | while read filename ; do echo "$filename" | grep "_" >/dev/null 2>&1 ; if [ $? -eq 0 ] ; then newfilename=`echo "$filename" | tr "_" " "`; mv "$filename" "$newfilename" ; echo "renamed \"$filename\" to \"$newfilename\"" ; fi ; done

List installed packages



How you do this will depend on the package manager used by your distribution. For RPM based distros, such as Red Hat, Fedora Core, Suse Linux, Cent OS, use:

rpm -qa | grep -i 'package-name'

The i switch on grep is pretty unnecessary really, as all libraries seem to use only lower case.

For .deb systems such as Debian and Ubuntu, use:

dpkg --list | grep -i 'package-name'