Some useful linux commands
How can I set a single .bashrc file for several users?
Output of man bash
(only a part):
FILES
/bin/bash
The bash executable
/etc/profile
The systemwide initialization file, executed for login shells
/etc/bash.bashrc
The systemwide per-interactive-shell startup file
/etc/bash.bash.logout
The systemwide login shell cleanup file, executed when a login shell exits
~/.bash_profile
The personal initialization file, executed for login shells
~/.bashrc
The individual per-interactive-shell startup file
~/.bash_logout
The individual login shell cleanup file, executed when a login shell exits
~/.inputrc
Individual readline initialization file
[Linux] 查詢 Linux 的 shutdown/reboot 記錄
last reboot # check boot log
last -x shutdown # check shutdown log
Why does su fail with “authentication error”? [duplicate]
What are the differences between “su”, “sudo -s”, “sudo -i”, “sudo su”?
sudo -s # ~/.bashrc is respected
sudo -i # ~/.bashrc is ignored
echo '<password>' | sudo -S <command>
Restart bash from terminal without restarting the terminal application (mac)?
exec bash
compgen -c
man <other-command>
which <other-command>
or
whereis <other-command> #used to locate binary, souce for a command
hostnamectl
uname -a
cat /etc/*-release
lsb_release -a
This can be installed by:
apt-get install -y lsb-release
Easy command line method to determine specific ARM architecture string?
dpkg --print-architecture
# amd64
one of them:
cat /etc/redhat-release
cat /etc/centos-release
cat /etc/system-release
cat /etc/os-release
rpm -qf /etc/redhat-release
less /etc/passwd
nvcc --version
or
nvcc -V
When nvcc
cannot be found:
cat /usr/local/cuda/version.txt
cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
hostnamectl status (for CentOS7)
systemd-detect-virt
facter | grep virtual (suitable for CentOS6)
lscpu
or:
cat /proc/cpuinfo
how to find out intel architecture family from command line
gcc -march=native -Q --help=target|grep march
Sample output:
-march= broadwell
How to know number of cores of a system in Linux?
echo $(nproc) # 4
The equivalent for Mac(nproc is not a command on Mac):
sysctl -n hw.logicalcpu
lsusb
nvidia-smi
If it says:
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.
Please also try adding directory that contains libnvidia-ml.so to your system PATH.
Following NVIDIA-SMI couldn't find libnvidia-ml.so library, add the following line to your ~/.bashrc
:
export LD_PRELOAD=/usr/lib/nvidia-418/libnvidia-ml.so
And then source ~/.bashrc
.
nvidia-smi --loop=1
nvidia-smi -l=1
or
nvidia-smi --loop-ms=500
nvidia-smi -lms=500
or
watch -n0 nvidia-smi
gpustat -cp
Note: -c for show command, -p for show pid
Install it by pip
.
nvidia-smi -q
First install peci1/nvidia-htop, and then:
nvidia-htop.py -c -l 50
This will show the processes' PID
, USER
, GPU
, MEM
, %CPU
, %MEM
, TIME
, COMMAND
.
How to find out shared memory and global memory size of GPU?
# cd <cuda_installatino_dir>/samples/1_Utilities/deviceQuery
cd /usr/local/cuda/samples/1_Utilities/deviceQuery
# compile the cpp file
make
# run deviceQuery
./deviceQuery
Result:
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GTX 1080"
CUDA Driver Version / Runtime Version 10.1 / 10.1
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 8116 MBytes (8510701568 bytes)
(20) Multiprocessors, (128) CUDA Cores/MP: 2560 CUDA Cores
GPU Max Clock rate: 1835 MHz (1.84 GHz)
Memory Clock rate: 5005 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 2097152 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS
NVIDIA NVML Driver/library version mismatch
lsmod | grep nvidia
rmmod <mod-name1> #follow the order from the last command
rmmod <mod-name2>
# rmmod ...
nvidia-smi # should work now
How do I kill all the computer processes shown in nvidia-smi?
kill -9 $(nvidia-smi | sed -n 's/|\s*[0-9]*\s*\([0-9]*\)\s*.*/\1/p' | sort | uniq | sed '/^$/d')
nvprof ./<cuda_executable>
glxinfo | grep -i device
Sample output:
Device: Mesa DRI Intel(R) Iris Plus Graphics 650 (Kaby Lake GT3e) (0x5927)
Note that it should be used on a computer with GUI, otherwise it gives the following error:
Error: unable to open display
Another option(can be used when there is no display): How do I find version of Intel graphics card drivers installed?
lspci -k | grep -EA3 'VGA|3D|Display'
Ref: Check your intel gpu usage via commandline
intel_gpu_top
clinfo --version
Sample output:
clinfo version 2.1.16.01.12
kill -9 `pgrep <your_exec_file_name>`
shutdown now
shutdown -r now
reference: https://help.ubuntu.com/community/BackupYourSystem/TAR
cd /
tar -cvpzf backup.tar.gz \
--exclude=/backup.tar.gz \
--exclude=/proc \
--exclude=/tmp \
--exclude=/mnt \
--exclude=/dev \
--exclude=/sys \
--exclude=/run \
--exclude=/media \
--exclude=/var/log \
--exclude=/var/cache/apt/archives \
--exclude=/usr/src/linux-headers* \
--exclude=/home/*/.gvfs \
--exclude=/home/*/.cache \
--exclude=/home/*/.local/share/Trash /
time <other-command>
Output:
real 1m4.743s
user 0m54.368s
sys 0m16.128s
locale -a
sudo locale-gen zh_TW.UTF-8
date
Get current time in seconds since the Epoch on Linux, Bash
date +%s
This is same as the following in Python:
import time
time.time()
cd /usr/share/zoneinfo
tzselect # and then follow the instructions
To make this change pernament, add the following line in the last of ~/.profile
:
TZ='Asia/Taipei'; export TZ
What does the sleep command do in Linux?
sleep <number_of_seconds>
cat /var/log/messages-<yyyymmdd> | less
eval $(<command-whose-output-is-the-commands-you-want-to-execute>)
printenv
env | grep -i "<env_var_name>"
export <env_var>="<value>"
Note that there should not be any spaces before or after '=', or it throws bad variable name
or bash: export: `=': not a valid identifier
.
Ref: How do I add environment variables?
echo $<env_var>"str_to_concat"
export PATH=$PATH:`pwd`
To make it work forever, add this line into the bottom of ~/.bashrc
and then source ~/.bashrc
.
unset <ENV_VAR_NAME>
hostname
ifconfig | grep -A 1 '<network-interface-name>' | tail -1 | tr -s ' ' | cut -d' ' -f 3 | cut -d':' -f 2
<network-card-name> could be enp1s0f1, eno1 or eth0 depending on your machine.
ip route get 1 | awk '{print $NF;exit}'
or
hostname -i
or
hostname -I | cut -d' ' -f1
To get external ip address:
How to Find the Public IP Address on a Linux System
host myip.opendns.com resolver1.opendns.com | grep "myip.opendns.com has" | awk '{print $4}'
or
dig +short myip.opendns.com @resolver1.opendns.com
How to Restart Network Interface in Linux
using ifdown:
ifdown <network-interface-name> && ifup <network-interface-name>
using ifconfig (cannot reconnect for me):
ifconfig <network-interface-name> down && ifconfig <network-interface-name> up
For Ubuntu, also:
sudo /etc/init.d/networking restart
or
sudo systemctl restart networking
Any way to find unused IP addresses on my network?
nmap -sP -PR 192.168.1.*
Ref to Configure static IP address on Ubuntu 16.04 LTS Server:
Edit the file /etc/network/interfaces
:
auto lo
iface lo inet loopback
auto <interface_name>
iface <interface_name> inet static
address <the_static_ip_address_to_be_set>
netmask 255.255.255.0
gateway <gateway>
dns-nameservers <gateway> 8.8.8.8 8.8.4.4
<interface_name>
can be found by ifconfig
, and <gateway>
can be found by route
.
Example:
auto lo
iface lo inet loopback
auto enp0s31f6
iface enp0s31f6 inet static
address 10.56.6.90
netmask 255.255.255.0
gateway 10.56.6.254
dns-nameservers 10.56.6.254 8.8.8.8 8.8.4.4
And then reboot.
How do I set my DNS when resolv.conf is being overwritten?
sudo vim /etc/resolvconf/resolv.conf.d/head
Add the following:
nameserver 8.8.8.8
nameserver 8.8.4.4
And then update:
sudo resolvconf -u
lsof -i -n -P
netstat -tulpn #--tcp|-t, --udp|-u, -l, --listening
#-p, --program: Show the PID and name of the program to which each socket belongs.
#-n, --numeric: Show numerical addresses instead of trying to determine symbolic host, port or user names.
nc <ip-address> <port-number> -v
Result:
Connection to <ip-address> <port-number> port [tcp/http] succeeded!
lsof -i:<port-number>
lsof -n -i4TCP:<port-number>
ssh <username>@<ip-address>
ssh <username>@<ip-address> -p<port-number>
ssh -t username@host '<your-command>'
Keep SSH session alive [closed]
Possible solution to packet_write_wait: Connection to xx.xx.xx.xx port 22: Broken pipe
.
Create ~/.ssh/config
, fill it with:
Host *
ServerAliveInterval 240
And then:
chmod 600 ~/.ssh/config
Error message:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Please contact your system administrator.
Add correct host key in C:\\Users\\xxx/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in C:\\Users\\xxx/.ssh/known_hosts:8
ECDSA host key for 192.168.1.17 has changed and you have requested strict checking.
Host key verification failed.
Solution:
ssh-keygen -R <ip-address>
sshpass -p <password> scp -r [email protected]:/some/remote/path /some/local/path
Ubuntu Linux: Start / Stop / Restart / Reload OpenSSH Server
sudo systemctl status ssh
or
sudo service ssh status
sudo systemctl enable ssh
How to list all enabled services from systemctl?
sudo systemctl list-unit-files | grep <service-name>
scp <username>@<ip-address>:/some/remote/directory/\{a,b,c\} ./
Linux (mv or cp) specific files from a text list of files?
xargs -a filenames.txt cp -t /destination/directory
Linux (mv or cp) specific files from a text list of files?
xargs -a filenames.txt mv -t /destination/directory
scp <username>@<ip-address>:/some/remote/directory/<original-name>.zip <new-name>.zip
rsync -P /local/file <username>@<ip-address>:/some/remote/directory
The -P
option shows the progress.
Copying only JPG from a directory structure to another location Linux
rsync -av --include='*.<filetype>' --include='*/' --exclude='*' --progress --delete --log-file=/home/<username>/$(date +%Y%m%d-%H%M%S)rsync.log /src/dir /dst/dir/dataBackup-$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S >> /home/<username>/rsync_time.txt # record the backup time
-a
: maintain directory structure
-v
: verbose
--progress
: show progress
--delete
: delete files deleted in /src/dir
--log-file
: save rsync result to a file
How to keep rsync from chown'ing transferred files?
rsync -av --no-o --no-g /src/dir /dst/dir
apt-cache search <package-name>
apt-cache show <package-name>
In /etc/apt/apt.conf
, add something like:
Acquire::http::proxy "http://child-prc/intel.com:913/";
Acquire::https::proxy "https://child-prc/intel.com:913/";
To unset, just clear /etc/apt/apt.conf
.
After changing the file, it will take effect immediately.
Set system proxy:
export http_proxy=http://child-prc.intel.com:913/
export https_proxy=https://child-prc.intel.com:913/
Unset system proxy:
unset http_proxy
unset https_proxy
In /etc/environment
, comment the following lines:
http_proxy="http://child-prc.intel.com:913/"
https_proxy="https://child-prc.intel.com:913/"
apt-get update
Issue with fetching http://deb.debian.org/debian/dists/jessie-updates/InRelease with docker
In Dockerfile, when using docker monsantoco/min-jessie:latest
, if met with :
W: Failed to fetch http://http.debian.net/debian/dists/jessie-updates/InRelease Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)
E: Some index files failed to download. They have been ignored, or old ones used instead.
This can be solved by adding:
printf "deb http://archive.debian.org/debian/ jessie main\ndeb-src http://archive.debian.org/debian/ jessie main\ndeb http://security.debian.org jessie/updates main\ndeb-src http://security.debian.org jessie/updates main" > /etc/apt/sources.list
in front of apt-get update
.
apt-get install -y <package-name>
What is the correct way to completely remove an application?
apt-get purge -y <package-name>
or
apt-get --purge autoremove <package-name>
apt-get install --download-only <package-name>
The .deb
file will be put in /var/cache/apt/archives
.
How to install a deb file, by dpkg -i or by apt?
dpkg -i /path/to/deb/file
apt install -f
error while rtabmapviz tries to load librtabmap_core.so
ldconfig
This solves: error while loading shared libraries...
.
ldconfig -p | grep <package-name>
apt list --installed
Determine destination location of apt-get install ?
dpkg -L <package-name>
yum list --showduplicates <package-name>
yum search <xxx>
ex: yum search java | grep 'java-1.8.0-openjdk-'
check total size of a directory, containing all its items(the unit is KB, MB or GB depending on its size)
du -sh <dirname>
How can I see a folder's exact size in bytes?
du -s -B1 <dirname>
du -h * | less
du -h --max-depth=1 <dirname> | sort -hr
or
du -h -d 1 <dirname> | sort -hr
sort: -h for recognizing K,M,G, -r for descending order
tree
or
tree </path/to/directory>
To list directory only:
tree -d
free # in KB
free -m # in MB
free -g # in GB
or
cat /proc/meminfo
df -h
df <dirname>
find <dirname> -name <filename> -print0 | du --files0-from=- -hc
This will show all matching files with its size in human-readable format.
it will do nothing if <dir-name> already exists, so we can always use this command without checking if a directory already exists!
mkdir -p <dir-name>
Add a User to a Group (or Second Group) on Linux
Linux Show All Members of a Group Command
getent group
Sample output:
root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:syslog,ubt
...
groups
Sample output:
ubt adm cdrom sudo dip plugdev lpadmin sambashare vboxsf
grep <groupname> /etc/group
Sample output:
vboxsf:x:999:ubt
sudo groupadd <groupname>
usermod -a -G <groupname> <username>
stat <filename>
Sample output:
File: 'faster-rcnn.pytorch/'
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 802h/2050d Inode: 120856675 Links: 10
Access: (0755/drwxr-xr-x) Uid: ( 1001/ user1) Gid: ( 0/ root)
Access: 2019-06-27 16:50:29.845551752 +0800
Modify: 2019-06-27 16:51:01.705798284 +0800
Change: 2019-06-27 16:51:01.705798284 +0800
Birth: -
How to find owner and group of a directory?
stat -c '%A' <filename>
Sample output:
drwxr-xr-x
stat -c "%U %G" <filename>
Sample output:
user1 root
stat -c "%s %n" <filename>
Sample output:
1091 README.md
-l for long listing format, -a for showing hidden files, -h for showing size in human readable format
ls -lah <filename>
Sample output:
total 212K
drwxr-xr-x 10 user1 root 4.0K Jun 27 16:51 .
drwxr-xr-x 9 user1 user1 4.0K Jun 27 16:40 ..
drwxr-xr-x 2 user1 root 4.0K Jun 26 10:37 cfgs
-rwxr-xr-x 1 user1 user1 1.1K Jun 26 11:11 class_names.pkl
The 3rd and 4th columns are the user and group of files respectly.
-l for long listing format, -d for only showing directories' names rather than their contents
ls -ld *
How can I sort the output of 'ls' by last modified date?
sorted by modification time, descending
ls -lt
sorted by modification time, ascending:
ls -ltr
Linux ls Command Sort Files By Size
ls -Slh # -S for --sort=size
ll <filename>
ls -1 <dirname>
ls -d <S>*/
or
echo <S>*/
ls -R <dirname>
How can I generate a list of files with their absolute path in Linux?
For directories (the / after ** is needed in bash to limit it to directories):
ls -d -1 "$PWD/"**/
For files and directories directly under the current directory, whose names contain a .:
ls -d -1 "$PWD/"*.*
For everything:
ls -d -1 "$PWD/"**/*
chmod [u,o,g,a][+,-,=][r,w,x] <filename>
chmod -R 777 <dirname>
chown -R username:groupname <dirname/*>
Only change owner:
chown -R username <dirname/*>
This solves the error of scp: /dirname/filename: Permission denied
, chmod
may also help.
top
# specify number of iterations as 1
top -n1
or
ps aux
to show full command in ps aux
:
Viewing full output of PS command
ps aux | less -+S
ps --sort=-pcpu
ps -o etime= -p "<pid>"
ps -p <pid> -o comm=
readlink -f /proc/<pid>/exe
kill -9 <pid>
Linux / Unix Find Command Avoid Permission Denied Messages
find . -name "<filename>" 2>/dev/null
How to stop the find command after first match?
find . ... -print -quit
How to combine 2 -name conditions in find?
find . \( -name "str1*" -o -name "str2*" \)
How can I generate a list of files with their absolute path in Linux?
find "$PWD" -name "<filename>"
find "$(pwd)" -name "<filename>"
find . -name "<filename>" -type f | xargs du -sh
Note that the double quotes around are important!
How to find only directories without subdirectories? [duplicate]
find . -type d -links 2 ! -empty
find . -type f -printf "%f\n"
How to get only names from find command without path
find . -name "*.jpg" -exec basename \{} .jpg \;
find . -type f ! -name "*.*"
find . -name "*.<file-type>" -type f -delete
or
find . -name "*.<file-type>" | xargs -i rm {}
or
find . -name "*.<file-type>" -exec rm {} \;
find . -type d -empty -delete
find . -name "<directory-name>" -type d -exec rm -rv {} \;
or
find . -name "<directory-name>" -type d -exec rm -rv {} +
find . -mindepth 1 -type d
find . -mindepth 1 -type d -exec rm -r "{}" \;
How to integrate mv command after find command?
find . -name "<filename>" -exec mv -t "<dst_path>" {} +
In Mac, where -t
is an illegal option to mv
:
find . -name "<filename>" -exec mv {} "<dst_path>" \;
find . -name "<filename>" -exec cp {} "<dst_path>" \;
find . -maxdepth 1 -name "<*word*>"
find . -maxdepth <depth> -mindepth <depth> -name "<*word*>"
find . -maxdepth 1 -print| awk -F/ ' length($NF)> <length> ' | xargs rm -rf
find <directory> -name "<*word*>"
find <directory> -name "<*word*>" -not -name "<*not-word*>"
How to exclude a directory in find . command
find / -name "<filename>" -not -path "./<the-path-to-be-excluded>*"
find / -iname "<filename>"
find / -name "<filename>" 2>/dev/null
find / -name "<filename>" | xargs -I{} dirname {}
find / -executable -type f
A directory is executable if one can cd into it. So one might only want to find executable "file".
Find files based on modified date(specifying the exact hour) [duplicate]
find . -newermt "<year1>-<month1>-<day1> 00:00:00" ! -newermt "<year2>-<month2>-<day2> 00:00:00"
How to recursively find the latest modified file in a directory?
find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "
To find files like 1567481018771_res.jpg or 1567481433382_res.jpg.
find . -regex "./[0-9]+_res\.jpg"
To find files like 1567481019363.jpg and 1567481189545.jpg.
find . -regex "./[0-9]+\.jpg"
find <prefix>* -name "*.jpg"
Find Files By Access, Modification Date / Time Under Linux or UNIX
To find a file modified less than 60 days:
find . -name "<filename>" -mtime -60 -print
How to report number of files in all subdirectories?
This will show full subdirectory name:
for f in *; do [ -d ./"$f" ] && find ./"$f" -maxdepth 1 -exec echo \; | wc -l && echo $f; done
Sample output:
72
$RECYCLE.BIN
17
8.9
3
System Volume Information
or
This won't show full subdirectory name, but now subdirectory name and its file count are in the same line:
for f in *; do [ -d ./"$f" ] && find ./"$f" -maxdepth 1 -exec echo \;|wc -l|tr '\n' ' ' && echo $f; done|awk '{print $2"\t"$1}'
Sample output:
$RECYCLE.BIN 72
8.9 17
System 3
How To Find Files Bigger Or Smaller Than X Size In Linux
find . -type f -size +100M
How to count total number of lines of found files?
This will show line counts for each file and finally total line count:
find . -name <pattern> -print0 | wc -l --files0-from=-
This will only show total line count:
find . -name <pattern> -exec cat {} + | wc -l
cat -n <filename>
cat > <filename> << EOF
<line1>
<line2>
<line3>
EOF
cat a.txt b.txt > new.txt
echo "<text>" > <filename>
printf '<line1>\n<line2>' > <filename>
If using echo
, \n
itself rather than a newline will be written to the file.
grep -n ".*<word>.*" <filename>
grep '.*<word>.*' -rnw <directory>/*.<file-type>
or
grep -rnw <directory>/*.<file-type> -e '.*<word>.*'
Grep regex NOT containing string
grep ".*<word>.*" -rnw . | grep -v ".*<word_to_exclude>.*"
Using grep to search for a string that has a dot in it
grep '<xxx\.xxx>' -rnw <directory>
grep <pattern> --include=\*.{type1, type2} -rnw <directory>
How to suppress binary file matching results in grep [closed]
grep <pattern> -Irn <directory>
# process a binary file as if it did not contain matching data
grep <pattern> --exclude=<file-to-be-excluded> -rnw <directory>
grep <pattern> --exclude-dir={<directory-to-be-excluded-1>,<directory-to-be-excluded-2>,<directory-to-be-excluded-3>} -rnw <directory>
grep <pattern> -rnw . | while IFS= read -r line ; do
> echo "$line" | cut -d'/' -f2
> done
Linux find and grep command together
find . -name '*fname*' -exec grep -H "text" {} \;
# -H: --with-filename
#show <b> lines before and <a> lines after the matched lines
<other-command> | grep <pattern> -B <b> -A <a>
#show <c> lines before and after the matched lines
<other-command> | grep <pattern> -C <c>
ex: docker run --help | grep runtime -C 5
grep -rnw --include=\*.<file-extension> ".*<word>.*"
ex: grep -rnw --include=*.sh ".SPARK." .
grep -i <pattern> -rnw .
grep <pattern> -rl .
How to find lines containing a string and then printing those specific lines and something else
grep <pattern> -rh .
How do I find files that do not contain a given string pattern?
grep <pattern> -rL .
grep - limit number of files read
grep <pattern> -r . | head -10
grep <^pattern> -r .
How to truncate long matching lines returned by grep or ack
grep ".{0,20}<pattern>.{0,20}" -oE .
egrep "<pattern-A>|<pattern-B>" <filename>
echo "hello world, it's a good day" | awk '{print$3}'
# it's
awk 'NR==<line_number>' <filename>
How to get the first column of every line from a CSV file?
awk -F" " '{print $1}' <filename>
Get last field using awk substr
split lines by "/" and get the last token
awk -F "/" '{print $NF}' <filename>
awk '{ print length }' <filename>
How to remove lines shorter than XY?
<other-command> | awk 'length($0) > <number>'
<other-command> | awk 'length($0) == <number>'
<other-command> | awk 'length($0) < <number>'
Exclude multiple columns using AWK
awk '{$1="";print}'
Split: how to split into different percentages?
The following split a file to 90% and 10%.
split -l $[ $(wc -l <filename>|cut -d" " -f1) * 90 / 100 ] <filename>
<other-command>| sort | uniq
Note that sort
is required.
sort command in unix with numeric sort
If one want to do numeric sort(sort as numbers), then use sort -V
.
Sort and count number of occurrence of lines
To also get their counts, use uniq -c
.
cp <filename>{,.bak}
How to move a file without preserving permissions
Following command works when /dst/dir is in CIFS.
sudo cp -f --no-preserve=mode,ownership /src/file /dst/dir
To escape multiple leading spaces, add \
before the first space.
sed -n <line_number>p <filename>
or
sed '<line_number>!d' <filename>
Delete specific line number(s) from a text file using sed?
sed -i -e '<start_line_number,end_line_number>d;<another_line_number>d' <filename>
sed -i 's/<regular expression>/<replacement>/g' <filename>
How to replace a string in multiple files in linux command line
To do this recursively:
!sed -i -e 's/<regular expression>/<replacement>/g' $(find . -name *.<file_extension>)
sed -n -i 'p;<line_number>a <line_to_insert>' <filename>
sed -i '/<line_to_match>/i <line_to_insert>' <filename>
sed -i '/<line_to_match>/a <line_to_insert>' <filename>
sed '<N> d' <filename>
How can I delete multiple random lines from a text file using sed?
shuf -i1-$(wc -l < <input_file>) -n<N> | sed 's/$/d/' | sed -f- <input_file> > <output_file>
sed -i 's/<word_to_delete>//' <filename>
How to delete a line with a given word using SED function
sed -i '/^<word>\b/d' <filename>
Input file:
a
a b
b a
ab
ad
b
Result file:
b a
ab
ad
b
Add a prefix string to beginning of each line
This will edit the file in-place:
sed -i -e "s/^/<prefix>/" <filename>
sed -i -e "s/$/<suffix>/" <filename>
To create a new file:
sed -e "s/^/<prefix>/" <filename> > <newfilename>
sed -e "s/$/<suffix>/" <filename> > <newfilename>
Example: add current directory as prefix to a file
How do I replace a token with the result of pwd
in sed?
sed -i -e "s?^?`pwd`/?" <filename>
sed -bi '/View.*\(.*\);/d' <filename>
-b
for binary mode, which prevents from changing windows line ending to that of linux.
-i
for inplace.
tr -s '<char>'
tr -d '<char>'
To use it with a file:
tr -d '<char>' < a.txt > tmp.txt && mv tmp.txt a.txt
How to concatenate multiple lines of output to one line?
tr '\r' ';'
This replace the Windows ^r
with ;
.
What is ^M
and how do I get rid of it?
dos2unix <filename> #support the wildcard sign *
What is a unix command for deleting the first N characters of a line?
cut -c 3-
Remove last character from line
This only works when all lines' ending characters are the same.
# -107.484, 56.0374, -3.34328)
# -79.6017, 21.7058, -1.92846)
cut -d "%" -f1
# -107.484, 56.0374, -3.34328
# -79.6017, 21.7058, -1.92846
cut -d '<delimiter>' -f <fields>
<fileds> could be 1 or "1,2,3".
cut -d '<delimiter>' -f <fields> <filename>
<other-command> | rev | cut -d' ' -f1 | rev
How to exclude last N columns in Bash on Linux?
<other-command> | rev | cut -d ' ' -f2- | rev
How to join multiple lines of file names into one with custom delimiter?
<other-command> | paste -sd "," -
-s
for serial, -d ","
for delimiter
remove particular characters from a variable using bash
<other-command> | tr -d '.'
<other-command> | tac
curl -X POST "<ip-address>:<port>/<subpage>" -d "<param1>=<value1>"
Method 1: wget
wget -O </directory/file_name> <your-url>
Method 2: curl
curl -o </directory/file_name> <your-url>
How to ignore invalid and self signed ssl connection errors with curl
curl https://<ip-address>:<port>/<subpage> --insecure
wget -P <dirname> <your-url>
wget -nc <your-url>
<some-command> | (while read x; do <other-command-using-$x>; done)
ex: ls *.json | (while read x; do echo $x; done)
export HISTTIMEFORMAT='%F %T '
history
method 1:
echo -en "$(<some-command-printing-non-ascii-characters>)"
method 2:
How to convert \uXXXX unicode to UTF-8 using console tools in *nix
apt-get install uni2ascii
<some-command-printing-non-ascii-characters> | ascii2uni -a U -q
<some-command> > <filename> #overwrite the file
<some-command> >> <filename> #append to the file
Shell重定向 &>file、2>&1、1>&2 、/dev/null的区别
<some-command> > <filename.txt> 2>&1
0, 1, 2 here means standard input, standard output and standard error respectively.
> <filename.txt>
means redirect standard output to the file.
2>&1
means redirect standard error to standard output.
How to redirect output to a file and stdout
apt-get install expect #install unbuffer
unbuffer <some-command> 2>&1 | tee <filename.txt>
<some-command> | wc -l
zip -r <foldername>.zip <foldername>
unzip troubleshooting: please check that you have transferred or created the zipfile in the appropriate BINARY mode and that you have compiled UnZip properly
zip -F <name>.zip -O <name>.fixed.zip
zip -FF <name>.zip -O <name>.fixed.zip
unzip <xxx.zip> -d </target/directory/to/unzip>
tar cf </packed/file/name>.tar </folder/name>
gzip -c </src/file/name> > </zipped/file/name>.gz
tar -czvf <xxx.tar.gz> <xxx>
tar -czf <parent.tar.gz> <parent> --exclude <parent>/<child1> --exclude <parent>/<child2>
Is there a way to see any tar progress per file?
tar cf - <xxx> -P | pv -s $(du -sb <xxx> | awk '{print $1}') | gzip > <xxx>.tar.gz
Mac OS:
tar cf - <xxx> -P | pv -s $(($(du -sk <xxx> | awk '{print $1}') * 1024)) | gzip > <xxx>.tar.gz
pv <xxx.tar.gz> | tar xzf - -C </target/directory/to/unzip>
tar -xvf <xxx.tar>
tar -xzvf <xxx.tar.gz>
tar -xzf <xxx.tar.gz> -C <xxx>
gunzip <xxx.gz>
or
gzip -d <xxx.gz>
7za -y x <xxx.7z>
bzip2 -dk <xxx.bz2>
How to Extract RAR Files in Ubuntu Linux
How to Open, Extract and Create RAR Files in Linux
extract .rar file, keep its structure:
unrar x <xxx.rar>
extract all its content to the same place:
unrar e <xxx.rar>
Unrar all file in directory without prompting
To disable the prompt, add -o+
option for automatic overriding.
cd -
dirs
Change directory and push it to directory stack:
pushd <dir>
Pop a directory from the stack and cd to it:
popd
diff <file1> <file2>
To ignore changes in spaces:
How can I diff 2 files while ignoring leading white space
diff <file1> <file2> -b
Given two directory trees, how can I find out which files differ?
diff -rq <dir1> <dir2>
-r
for recursive, -q
for short mode.
diff -x "<pattern>" -r -q <dir1> <dir2>
cmp <file1> <file2>
Sample output:
<file1> <file2> differ: byte 132161537, line 437952
cmp -l <file1> <file2>
Sample output:
132161537 274 0
132161538 4 0
132161539 225 0
132161540 212 0
132161541 261 0
...
compare executable or object file
diff <(objdump -d <file1>) <(objdump -d <file2>)
tail -n <num_lines> <filename.txt>
+G performs the action of scrolling to bottom automatically
<other-command> | less +G
run your command in background, write its output to a text file, and watch that text file simultaneously
nohup <some-command> </dev/null &><filename.txt> &
tail -f <filename.txt>
ln -s <target> <linkname>
The <target>
should be full path, otherwise there will be an error: Too many levels of symbolic links
.
Difference of hard and soft link can be found in: Linux ln command.
To check if it successes, use:
ls -l <linkname>
If it successes, it will output something like:
lrwxrwxrwx 1 root root 50 四 29 10:21 Eigen -> /home/ubt/Documents/installation/eigen-3.3.4/Eigen
in blue.
Otherwise it will output something like:
total 0
lrwxrwxrwx 1 root root 41 四 29 10:18 Eigen -> /Documents/installation/eigen-3.3.4/Eigen
in red.
readlink <linkname>
Reference to: How to Remove a Symbolic Link (Symlink)
unlink <linkname>
or
rm -rf <linkname>
mkdir <dir2>
mount --bind /src/dir /dst/dir
--bind makes it possible to remount part of the file hierarchy somewhere else, and it solves the error:
mount: /src/dir is not a block devicere
In VirtualBox's VM, assume /media/sf_implementation
is a directory shared from host. Use the following to create a binding from /home/ubt/Documents/implementation
to /media/sf_implementation
:
mkdir /home/ubt/Documents/implementation
cd /media
sudo mount -t vboxsf implementation /home/ubt/Documents/implementation
mount -t cifs -o vers=1.0,user=<cifs_user_name>,password=<cifs_user_password> "//<ip_addr>/src/dir" /dst/dir
To specify the owner of the mount point, ref: How to chown a file on a CIFS mount?:
-o uid=<linux_user_name>
umount /dst/dir
Why does my NTFS partition mount as read only?
sudo ntfsfix /dev/sdb1
And then umount and mount it again.
readlink -e <linkname>
readlink -f returns something even for non-existing <linkname>
readlink -e will return null for non-existing <linkname>
jar tf <xxx.jar>
or
jar tvf <xxx.jar>
less <xxx.zip>
find the difference set of <file1> - <file2>
comm -23 <(sort <file1>) <(sort <file2>)
find the intersection of <file1> - <file2>
comm -12 <(sort <file1>) <(sort <file2>)
Note that '<' and '(' should not be separated, otherwise there will be a bash: syntax error near unexpected token `('
!
More details: Linux comm command brief tutorial
bash, Linux: Set difference between two text files
ab -n <total-requests-count> -c <concurrent-requests-count> <url>
send <total-requests-count> requests to <url> in which of them are concurrent.
sudo service lightdm stop
pass output as an argument for cp in bash [duplicate]
cp `ls /source/directory` /target/directory
rm `ls /dirname/filename`
Argument list too long when copying files
for i in *.jpg; do cp "$i" /target/directory; done
find /src/dir -type f -name "*.<filetype>" -exec mv {} /dst/dir \;
Use the following to replace ls /dir/name/*.jpg | wc -l
ls -l /dir/name | fgrep .jpg | wc -l
alias
alias <dst_excutable>=<src_executable>
e.g.
alias python=python3
how can I rename multiple files by inserting a character?
Rename file with extension .cloud
to _cloud.cpp
.
rename 's/.cloud$/_cloud.cpp/' *.cloud
Rename: delete something like -0_1
, -1_2
from file names:
rename 's/-[0-9]_[0-9]//' *
Rename: add and remove .bak:
rename 's/$/.bak/' *
rename 's/.bak//' *
Or using for loop and mv:
for f in $(ls *.xml); do
mv "$f" "cam2_$f"
done
This replace ./FP32/b4/benchmark_CPU__.txt
to ./FP32/b4/benchmark_CPU_t1_s1.txt
:
for fname in $(grep FPS -rl . --include=*__.txt)
do
mv "$fname" "${fname/__/_t1_s1}"
done
paste <filename1> <filename2> ... <filenameN> | expand -t <column_width>
The above requires one to find <column_width>
manually.
The following find the suitable <column_width>
automatically.
paste <filename1> <filename2> ... <filenameN> | expand -t $(( $(wc -L < <filename1>) + 2 ))
shuf <filename>
How to randomly sample a subset of a file
Sample <count>
lines from <filename>
.
shuf -n <count> <filename>
shuf -n <count> <filename> -o <filename>
Where do I find the core dump in ubuntu 16.04LTS?
Core dumped, but core file is not in the current directory?
ulimit -c unlimited
echo "core.%e.%p" > /proc/sys/kernel/core_pattern
sudo service apport start
Note: if you are inside a docker container, do the above on the HOST!
How do I analyze a program's core dump file with GDB when it has command-line parameters?
gdb <executable> <core_file>
or
gdb <executable> -c <core_file>
view <filename>
Check if a directory exists in a shell script
if [ -d "$DIRECTORY" ]; then
# Control will enter here if $DIRECTORY exists.
fi
if [ ! -d "$DIRECTORY" ]; then
# Control will enter here if $DIRECTORY doesn't exist.
fi
for OUTPUT in $(<other-command>)
do
command1 on $OUTPUT
command2 on $OUTPUT
commandN
done
Bash: Continue In a For / While Loop
for i in 1 2 3 4 5 6
do
### just skip printing $i; if it is 3 or 6 ###
if [ $i -eq 3 -o $i -eq 6 ]
then
continue ### resumes iteration of an enclosing for loop ###
fi
# print $i
echo "$i"
done
How can I pass a command line argument into a shell script?
In your script, use "$1"
, "$2"
, ... to represent the first, second, ... arguments.
Propagate all arguments in a bash shell script
To pass all arguments at one time, use "$@"
, for example, writing a script file named compile.sh
with following content:
g++ "$@" -I /usr/local/include/opencv4 -L /usr/local/lib -lopencv_imgcodecs -lopencv_core -lopencv_imgproc -lopencv_highgui -std=c++11
And by using:
./compile.sh a.cpp b.cpp c.cpp
All a.cpp
, b.cpp
, c.cpp
will be the argument of g++
.
how to recognize video codec of a file with ffmpeg
mediainfo --Inform="Video;%Codec%" <input_video>
# AVC
ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 <input_video>
# h264
How to retrieve video file information from command line under Linux?
ffprobe -v error -show_format -show_streams <input_video>
Converting .mp4 to .264 using FFmpeg
ffmpeg -i <input_mp4_video> -an -vcodec libx264 -crf 23 <output_h264_video>
First edit your file:
nano xxx.txt
Then press Ctrl + O to save, press Enter to confirm, and finally press Ctrl + X to exit.
Emacs Basics Save: Ctrl + x Ctrl + s
Quit: Ctrl + x Ctrl + c
P.S. This also works for Mac, no need to replace Ctrl with Command.
How to open a folder in linux via terminal? [closed]
nautilus <folder_name>
How to open text, video, or picture files via terminal
xdg-open <filename>
Ubuntu (GNOME) default video player name
Find in launcher configuration file:
grep -r Videos /usr/share/applications
Sample output:
/usr/share/applications/org.gnome.Totem.desktop:Name=Videos
/usr/share/applications/totem.desktop:Name=Videos
Find out which package supplies this file:
dpkg -S /usr/share/applications/org.gnome.Totem.desktop
dpkg -S /usr/share/applications/totem.desktop
Sample output:
totem: /usr/share/applications/org.gnome.Totem.desktop
totem: /usr/share/applications/totem.desktop
So the default video player is totem
.
evince xxx.pdf
How to create animated GIF images of a screencast?
sudo apt install -y peek
peek