Home » Programming

Bash

Scripting with Bourne-Again SHell...

renaming non-ascii file names

Submitted by sandip on Mon, 08/14/2017 - 12:52

Below script replaces non-ascii character with an underscore in file/folder names.

Before running the below script, cd to the folder with files that need renamed first.

#!/bin/bash
# rename2ascii.sh
# Replaces non-ascii character with an underscore.

function rename_to_ascii() {
  echo -n "$1" | \
  perl -ne '$new = $_; if($new =~ s/[^[:ascii:]]/_/g) {
    print("Renaming $_ to $new\n");
    rename($_, $new);
  }'
}

export -f rename_to_ascii

find -depth -exec bash -c 'rename_to_ascii "$0"' {} \;

»

Handling filenames with spaces, carriage returns or other control characters

Submitted by sandip on Mon, 08/27/2012 - 23:42

find -print0 | while IFS= read -rd $'\0' filename ; do echo "[$filename]" ; done

-print0, prints the full file name on the standard output, followed by a null character instead of the newline character.

IFS, is the "Internal Field Separator" that is used for word splitting after expansion. Here, IFS is set to null string.

-r, specifies that backslash "\" does not act as an escape character.

-d, is the delimeter. Which in this case is the null character '\0'.

$'\0', the $ prefixed single quoted string decodes the backslash escape character. In this case a null character.

[], is simply there to print out the text, so you notice any spaces in the beginning and end of text.

»

redirect stdout/stderr within bash script

Submitted by sandip on Fri, 06/22/2012 - 08:04

This executes the date command via shell script and logs it to the specified file with current pid.

#!/bin/bash
LOG=$$.log
exec > $LOG 2>&1
date

»

check webpage load time via wget

Submitted by sandip on Tue, 02/22/2011 - 17:19

Here is a simple one liner to check on download time of a webpage:

(time wget -p --no-cache --delete-after www.linuxweblog.com -q ) 2>&1 | awk '/real/ {print $2}'

»

Domain searching with bash

Submitted by sandip on Thu, 01/13/2011 - 17:21

Below is a simple bash script that uses the dictionary to go through 3 letter words and does a whois check to see if the domain is available. You probably won't find any of those as all would have been taken. May want to try with 5 or 6 letters instead, in which case change the value of "$count".

#!/bin/bash
# domain_search.sh

# Get a list of 3 letter domains.
for x in `cat /usr/share/dict/words`; do count=`echo $x| wc -m`; [ $count = 4 ] && echo $x; done > domains_list.txt

# Get whois record.
for x in `cat ./domains_list.txt` ; do (whois $x.com | grep -q '^No match for domain') && echo $x; sleep 60; done > domains_available.txt

# Change to lowercase and sort print.
cat domains_available.txt | tr [:upper:] [:lower:]|sort| uniq

»

Clean up config files

Submitted by sandip on Tue, 10/12/2010 - 21:53

Below are one liners to clean out all comment and blank lines with grep and sed, usually in config files.

grep -v "^#\|^$" <conf_file>

or

grep -v "^\#" <conf_file> | sed '/^$/d'

»

Check for entries in passwd file

Submitted by sandip on Mon, 05/10/2010 - 00:51

Check for entries in passwd file:

getent passwd username >/dev/null 2>&1 && echo "user exists" || echo "user does not exist"

id username >/dev/null 2>&1 && echo "user exists" || echo "user does not exist"

awk -F':' '$1 ~ /^username$/ {print $0}' /etc/passwd  | grep -w username >/dev/null 2>&1 && echo "user exists" || echo "user does not exist"

»

Search bot report

Submitted by sandip on Mon, 03/22/2010 - 13:41

Here is a simple bash script to get a daily report of search bot results of Success (200) and Failed (404) hits:

#!/bin/bash
# bot_report.sh
# usage: ./bot_report.sh [botName] [logPath]
# default: ./bot_report.sh Googlebot /var/log/httpd/access_log

######################################
# Run this in a daily cron           #
# 59 23 * * * /path/to/bot_report.sh #
######################################

# Commands
GREP=/bin/grep
DATE=/bin/date
MKDIR=/bin/mkdir
AWK=/bin/awk
SORT=/bin/sort
UNIQ=/usr/bin/uniq
TMPWATCH=/usr/sbin/tmpwatch
CAT=/bin/cat
MAIL=/bin/mail
ECHO=/bin/echo

# Global Variables
DEFAULT_BOT_NAME=Googlebot
DEFAULT_LOG_FILE=/var/log/httpd/access_log
if [ -z "$1" ]; then
BOT_NAME=${DEFAULT_BOT_NAME}
else
BOT_NAME=$1
fi
if [ -z "$2" ]; then
LOG_FILE=${DEFAULT_LOG_FILE}
else
LOG_FILE=$2
fi
PREFIX_LOG=`$ECHO $LOG_FILE | sed 's/\//_/g'`
TMP_LOG_PATH=/tmp/bot_report
TMP_LOG_FILE=${TMP_LOG_PATH}/${PREFIX_LOG}_`${DATE} +%F`.log
TMP_REPORT_FILE=${TMP_LOG_PATH}/${BOT_NAME}_report.txt
EMAIL=user@domain.tld

###########################
# Nothing to change below #
###########################

# Produce a temp file to work with for todays date
tmp_file_out() {
[ -d "${TMP_LOG_PATH}" ] || $MKDIR ${TMP_LOG_PATH}
if [ ! -f "${TMP_LOG_FILE}" ]; then
$GREP `$DATE +%d/%b/%Y` $LOG_FILE > $TMP_LOG_FILE
fi
}

# Clean up temp file older than a day
tmp_file_clean() {
$TMPWATCH 24 $TMP_LOG_PATH
}

# Create report
report_out() {
$ECHO "######## Success Hits ########" > $TMP_REPORT_FILE
$GREP " 200 " $TMP_LOG_FILE | $AWK -v bot="$BOT_NAME" '$0 ~ bot {print $7}' | $SORT | $UNIQ -c | $SORT -rn >> $TMP_REPORT_FILE
$ECHO >> $TMP_REPORT_FILE
$ECHO "######## Failed Hits ########" >> $TMP_REPORT_FILE
$GREP " 404 " $TMP_LOG_FILE | $AWK -v bot="$BOT_NAME" '$0 ~ bot {print $7}' | $SORT | $UNIQ -c | $SORT -rn >> $TMP_REPORT_FILE
}

# Mail report
mail_report() {
$CAT $TMP_REPORT_FILE | $MAIL -s "bot report: ${BOT_NAME}" $EMAIL
}

#
# Main
#
tmp_file_out
report_out
mail_report
tmp_file_clean

exit 0

»

svn checkout via shell script

Submitted by sandip on Mon, 03/30/2009 - 23:19

Very often development servers only have self-signed certificate for ssl connection. I've recently had to create a script that would checkout from a https svn repository that would fail with "Server certificate verification failed: issuer is not trusted..." . Below is a workaround used to temporarily trust the certificate.

svn --username $SVN_USER --password $SVN_PASS --no-auth-cache checkout ${REPO_URL}/${REPO_PATH} $REPO_PATH <<EOF 2>/dev/null
t
EOF

»

Week of Month

Submitted by sandip on Thu, 01/08/2009 - 10:24

Here is a simple one liner to get the week of month via awk from a `cal` output:

$ cal | awk -v date="`date +%d`" '{ for( i=1; i <= NF ; i++ ) if ($i==date) { print FNR-2} }'

»

Syndicate content
Comment