AWK

I noticed a bot scraping using fake GoogleBot useragent string.

Here is a one liner that can detect the IPs to ban:

It does a case-insensitive awk search for keyword "googlebot" from apache log file removing IPs with "66.249.71." which belongs to google and prints the output in a sorted hit count.

You can validate the IPs with:

Replace the IP value with the one you want to check.

Below will list all of the files that are used for apache authentication in /var/www/html file path:

Here is the breakdown:

Find all files named ".htaccess" at path "/var/www/html"

The search output gets piped via xargs to awk, deleting leading whitespace (spaces and tabs) from front of each line and output is of only the second field of lines containing the text "File".

Awk output is further piped through sort and uniq which results in the files being used for apache authentication.

The content of the input file becomes stdin for rm and awk. rm ignores the input and removes the file, but its file descriptor remains open until both commands, rm and awk is complete. AWK process this "nameless" file and creates a new file:

• "-v" sets the awk variable that is passed in via shell script variable.
  
• "gsub" replaces with "" all occurrence of & in the 7th field. Use "sub" for single/first occurrence substitution or GNU Awk's gensub for more articulated substitutions.
  
• "1" is a shortcut which means print the current record:

Here is a simple one liner to get the week of month via awk from a `cal` output: