Unix scripting notes

Collection of everyday snippets of things I find myself googling for way too often.

shell

Shebang

#!/bin/bash

Compare values:

if [ "$a" -gt "$b" ]; then
    ...
fi

Write something into a file

 echo "some data for the file" >> fileName.txt

Go through every directory in a directory

for d in */ ; do
     echo "$d"
done

Check if file exists

FILE=/etc/resolv.conf
if [ -f "$FILE" ]; then
    echo "$FILE exists."
fi

Go through every line in a file

while IFS="" read -r p || [ -n "$p" ]
do
  printf '%s\n' "$p"
done < peptides.txt

Print out current time:

now=$(date +"%T")
echo "Current time : $now"

Output:

Current time : 13:31:55

One line for loop

for i in *; do echo $i; done

One line while loop

while true; do foo; sleep 2; done

Increment a variable

var=$((var+1))

If line exists in a file

if grep -q "${tikarray[$vidrandom]}" playlog.txt;
    then
        #something
    else
        #something else
    fi

Bash Shell Find Out If a Variable Is Empty Or Not

if [ -z "$var" ]
then
      echo "\$var is empty"
else
      echo "\$var is NOT empty"
fi

Replace unwanted characters with _ in string

sed -r 's/[<>:"/\|?* ]+/_/g'

How many files in directory, recursive

find <directory> -type f | wc -l

While loop until output does or does not contain a string

while true; do
  str=`date +%S`
  echo Output: $str
  # Use the below when you want the output not to contain some string
  if [[ ! $str =~ 5 ]]; then
  # Use the below when you want the output to contain some string
  # if [[ $str =~ 7 ]]; then
    break
  fi
  sleep .5
done
echo Finished: `date`

---

wget

Generic wget snippet to download things

wget -c -r -np --tries=inf -R "index.html*" -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" <link>

Wget output into a file foo.html

wget -O foo.html google.com

Wget user agent

wget -U "User Agent Here" "http://website.url.here"

example google chrome

wget -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" "http://website.url.here"

List of possible user agents:
https://www.networkinghowtos.com/howto/common-user-agent-list/

Get webpage, save into file and grep that

wget "https://www.nettiauto.com/bmw/i3?id_vehicle_type=1&id_country[]=73" -qO temp.txt && cat temp.txt | grep totAdInList

Output:
<h2 class="ma0">Vaihtoautohaun tulokset: <span class="totAdInList">66</span> vaihtoautoa.</h2>

Output into variable

RESULT=$(wget -qO- http://example.com)
echo $RESULT

---

grepping and formatting

grep only lines that say FOO

grep FOO

Replace THING with a newline

sed 's/THING/\n/g'

Remove everything before FOO and replace with BAR

sed 's/^.*FOO/BAR/'

Take only the first column

awk '{print $1}'

Remove file extensions

loadfile=${loadfile%%.*}

Remove four first lines

tail -n +4 file.txt

Remove first three characters

cut -c 3-

Only take the first three characters

cut -c -3

Divide by “/” and only take the first field.

cut -f1 -d"/"

Replace spaces with _ in a variable

str="${str// /_}"

Get uptime in short format

uptime -p | sed 's/up\s*//g' | sed 's/\s*days/d/g' | sed 's/\s*hours/h/g' | sed 's/\s*minutes/m/g' | sed 's/,//g'

Produces output like this:
4d 2h 19m