Random Tech Ramblings

(This is a work in progress) Get the average of a column ls -l | grep -v total | awk -F' ' '{sum += $5 } END { print "AVG=", sum/NR }' Print a file with line numbers awk '{ print FNR "\t" $0 }' file Print number of lines in a file (analogous to wc -l ) awk 'END { print NR }'

Using netstat PORT=80; sudo netstat -ltnp | grep ":${PORT}" Using lsof PORT=80; sudo lsof -i :${PORT}

PORT=5432; netstat -an |grep ":${PORT} " | \ awk '{count[$5]++} END {for (host in count) print host, count[host]}' To view the same over time: PORT=5432; watch "netstat -an |grep \":${PORT} \" | \ awk '{count[\$5]++} END {for (host in count) print host, count[host]}'"

If you have unexplained disk usage that isn't reflected in du output, you most likely have a process holding onto a filehandle for a file that has since been deleted. You can use lsof to find the process/file by running: lsof -nP | grep '(deleted)'

tail -f /path/to/log/file | pv -l -i 10 -r > /dev/null pv options user: -l count lines -i10 refresh every 10 seconds -r display rate counter

Logging Filter by request host header varnishlog -q 'ReqHeader ~ "Host: example.com"' Filter by request url varnishlog -q 'ReqURL ~ "^/some/path"' Filter by client IP (behind reverse proxy) varnishlog -q 'ReqHeader ~ "X-Real-IP: .*123.123.123.123"' Filter by request host header and show request URL and referrer header varnishlog -q 'ReqHeader ~ "Host: (www\.)?example\.com"' -i "ReqURL" -I "ReqHeader:Referrer:" Admin Purge Cache varnishadm 'ban req.url ~ "^/some/path/"' Show VCLs varnishadm vcl.list Show backends varnishadm backend.list

While trying to reproduce an issue, it's often useful to be able to artificially introduce latency without simply killing a service. The tc command, coupled with iptables allows you to achieve that goal. Run all of the below as root. All network traffic to/from a host Add 10ms to all network activity: tc qdisc add dev eth0 root netem delay 10ms Show config: tc -s qdisc Undo it: tc qdisc del dev eth0 root netem Specific flows Add 10ms to all HTTP responses from a webserver running on port 80 on this host: tc qdisc add dev eth0 root handle 1: prio priomap 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 tc qdisc add dev eth0 parent 1:2 handle 20: netem delay 10ms tc filter add dev eth0 parent 1:0 protocol ip u32 match ip sport 80 0xffff flowid 1:2 Undo it: tc qdisc del dev eth0 root handle 1: prio priomap 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

nstat is a tool for collecting linux network statistics. I use it in my monitoring stack to collect information about UDP data loss. The command maintains state, and on rare occasions, that state becomes corrupt. Corrupted state results in the command dumping core without any other explanation. $ nstat Aborted (core dumped) An strace of the command is similarly unhelpful. The best way I've found to rectify this situation is to reset the command's history nstat -r

Install pktstat sudo apt-get install pktstat Show all UDP traffic to port 8125, ordered by quantity of data sudo pktstat -tn udp dst port 8125

Assuming the AWS command is installed and configured correctly AWS_ACCESS_KEY=AKIAXXXXXXXXEXAMPLE aws --output text iam list-users | awk '{print $NF}' | xargs -P10 -n1 aws --output text iam list-access-keys --user-name | grep ${AWS_ACCESS_KEY}

Remote file: curl -L -s http://path/to/remote/file | shasum -a 256 | cut -c-12 Local file: shasum -a 256 /path/to/file | cut -c-12

Sometimes you want to redirect the output of a command into a file that you don't own. $ ls -al /tmp/root_owns_this -rw-r----- 1 root wheel 10 Aug 21 09:06 /tmp/root_owns_this $ echo "test" > /tmp/root_owns_this -bash: /tmp/root_owns_this: Permission denied Your first instinct may be to simply use sudo: $ sudo echo "test" > /tmp/root_owns_this -bash: /tmp/root_owns_this: Permission denied Same error. Why? Because sudo is only elevating the privileges of the echo command. The redirect still belongs to your shell. I'm aware of two options to get around this, although I'm sure there are others. $ echo "test" | sudo tee /tmp/root_owns_this # Create/overwrite the file test $ echo "test" | sudo tee -a /tmp/root_owns_this # Append to the file test or $ sudo bash -c "echo \"test\" > /tmp/root_owns_this" # Create/overwrite the file $ sudo bash -c "echo \"test\" >> /tmp/root_owns_this # Appe...