Category: Daily Hacks

Hack #21 -> Sed Basics – Find and Replace Using RegEx

This hack explains how to use sed substitute command “s”.

The `s’ command is probably the most important in `sed’ and has a lot

of different options.

The `s’ command attempts to match the pattern space against the

supplied REGEXP; if the match is successful, then that portion of the

pattern space which was matched is replaced with REPLACEMENT.

Syntax:

 

#sed ‘ADDRESSs/REGEXP/REPLACEMENT/FLAGS’ filename

#sed ‘PATTERNs/REGEXP/REPLACEMENT/FLAGS’ filename

 

  • s is substitute command
  • / is a delimiter
  • REGEXP is regular expression to match
  • REPLACEMENT is a value to replace

FLAGS can be any of the following :

  • g Replace all the instance of REGEXP with REPLACEMENT
  • n Could be any number,replace nth instance of the REGEXP with

REPLACEMENT.

  • p If substitution was made, then prints the new pattern space.
  • i match REGEXP in a case-insensitive manner.
  • w file If substitution was made, write out the result to the given

file.

  • We can use different delimiters ( one of @ % ; : ) instead of /

Let us first create thegeekstuff.txt file that will be used in all the

examples mentioned below.

 

$ cat thegeekstuff.txt

 

# Instruction Guides

  1. Linux Sysadmin, Linux Scripting etc.
  2. Databases – Oracle, mySQL etc.
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux
  5. Productivity (Too many technologies to explore, not

much time available)

# Additional FAQS

  1. Windows- Sysadmin, reboot etc.

 

Substitute Word “Linux” to “Linux-Unix” Using sed s//

In the example below, in the output line “1. Linux-Unix Sysadmin, Linux

Scripting etc” only first Linux is replaced by Linux-Unix. If no flags are

specified the first match of line is replaced.

 

$ sed ‘s/Linux/Linux-Unix/’ thegeekstuff.txt

 

# Instruction Guides

  1. Linux-Unix Sysadmin, Linux Scripting etc.
  2. Databases – Oracle, mySQL etc.
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux-Unix
  5. Productivity (Too many technologies to explore, not

much time available)

# Additional FAQS

  1. Windows- Sysadmin, reboot etc.

 

Substitute all Appearances of a Word Using sed s//g

The below sed command replaces all occurrences of Linux to Linux-Unix

using global substitution flag “g”.

 

$ sed ‘s/Linux/Linux-Unix/g’ thegeekstuff.txt

 

# Instruction Guides

  1. Linux-Unix Sysadmin, Linux-Unix Scripting etc.
  2. Databases – Oracle, mySQL etc.
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux-Unix
  5. Productivity (Too many technologies to explore, not

much time available)

# Additional FAQS

  1. Windows- Sysadmin, reboot etc.

 

Substitute Only 2nd Occurrence of a Word Using sed s//2

In the example below, in the output line “1. Linux Sysadmin, Linux-Unix

Scripting etc.” only 2nd occurrence of Linux is replaced by Linux-Unix.

 

$ sed ‘s/Linux/Linux-Unix/2’ thegeekstuff.txt

 

# Instruction Guides

  1. Linux Sysadmin, Linux-Unix Scripting etc.
  2. Databases – Oracle, mySQL etc.
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux
  5. Productivity (Too many technologies to explore, not

much time available)

# Additional FAQS

  1. Windows- Sysadmin, reboot etc.
  2. Write Changes to a File and Print the Changes Using sed

s//gpw

The example below has substitution with three flags. It substitutes all

the occurrence of Linux to Linux-Unix and prints the substituted output

as well as written the same to the given the file.

 

$ sed -n ‘s/Linux/Linux-Unix/gpw output’ thegeekstuff.txt

 

  1. Linux-Unix Sysadmin, Linux-Unix Scripting etc.
  2. Storage in Linux-Unix

 

$ cat output

 

  1. Linux-Unix Sysadmin, Linux-Unix Scripting etc.
  2. Storage in Linux-Unix
  3. Substitute Only When the Line Matches with the Pattern

Using sed

In this example, if the line matches with the pattern “-”, then it replaces

all the characters from “-” with the empty.

 

$ sed ‘/\-/s/\-.*//g’ thegeekstuff.txt

 

# Instruction Guides

  1. Linux Sysadmin, Linux Scripting etc.
  2. Databases
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux
  5. Productivity (Too many technologies to explore, not

much time available)

# Additional FAQS

  1. Windows

 

Delete Last X Number of Characters From Each Line Using

sed

 

This sed example deletes last 3 characters from each line.

$ sed ‘s/…$//’ thegeekstuff.txt

 

# Instruction Gui

  1. Linux Sysadmin, Linux Scripting e
  2. Databases – Oracle, mySQL e
  3. Security (Firewall, Network, Online Security e
  4. Storage in Li
  5. Productivity (Too many technologies to explore, not

much time availab

# Additional F

  1. Windows- Sysadmin, reboot e

 

Eliminate Comments Using sed

Delete all the comment lines from a file as shown below using sed

command.

$ sed -e ‘s/#.*//’ thegeekstuff.txt

 

  1. Linux Sysadmin, Linux Scripting etc.
  2. Databases – Oracle, mySQL etc.
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux
  5. Productivity (Too many technologies to explore, not

much time available)

  1. Windows- Sysadmin, reboot etc.
  2. Eliminate Comments and Empty Lines Using sed

 

In the following example, there are two commands separated by ‘;’

First command replaces the lines starting with the # to the blank lines

Second command deletes the empty lines.

 

$ sed -e ‘s/#.*//;/^$/d’ thegeekstuff.txt

 

  1. Linux Sysadmin, Linux Scripting etc.
  2. Databases – Oracle, mySQL etc.
  3. Security (Firewall, Network, Online Security etc)
  4. Storage in Linux
  5. Productivity (Too many technologies to explore, not

much time available)

  1. Windows- Sysadmin, reboot etc.
  2. Convert DOS newlines (CR/LF) to Unix format Using sed

 

Eliminate HTML Tags from file Using sed

 

In this example, the regular expression given in the sed command

matches the html tags and replaces with the empty.

$ sed -e ‘s/<[^>]*>//g’

 

This <b> is </b> an <i>example</i>.

This is an example

Advertisement

Hack #20 -> Execute Commands in the Background

You can use one of the 5 methods explained in this hack to execute a

Linux command, or shell script in the background.

 

Method 1. Use &

You can execute a command (or shell script) as a background job by

appending an ampersand to the command as shown below.

 

$ ./my-shell-script.sh &

 

Method 2. Nohup

After you execute a command (or shell script) in the background using

&, if you logout from the session, the command will get killed. To avoid

that, you should use nohup as shown below.

 

$ nohup ./my-shell-script.sh &

 

Method 3. Screen Command

After you execute a command in the background using nohup and &, the

command will get executed even after you logout. But, you cannot

connect to the same session again to see exactly what is happening on

the screen. To do that, you should use screen command.

Linux screen command offers the ability to detach a session that is

running some process, and then attach it at a later time. When you

reattach the session later, your terminals will be there exactly in the way

you left them earlier.

 

Method 4. At Command

Using at command you can schedule a job to run at a particular date

and time. For example, to execute the backup script at 10 a.m

tomorrow, do the following.

 

$ at -f backup.sh 10 am tomorrow

 

Method 5. Watch Command

To execute a command continuously at a certain interval, use watch

command as shown below.

 

$ watch df -h

Hack #19 -> Display Total Connect Time of Users

Ac command will display the statistics about the user’s connect time.

 

Connect time for the current logged in user

With the option –d, it will break down the output for the individual days.

In this example, I’ve been logged in to the system for more than 6 hours

today. On Dec 1st, I was logged in for about 1 hour.

$ ac –d

Dec 1 total 1.08

Dec 2 total 0.99

Dec 3 total 3.39

Dec 4 total 4.50

Today total 6.10

 

Connect time for all the users

To display connect time for all the users use –p as shown below. Please

note that this indicates the cumulative connect time for the individual

users.

$ ac -p

john 3.64

madison 0.06

sanjay 88.17

nisha 105.92

ramesh 111.42

total 309.21

 

Connect time for a specific user

To get a connect time report for a specific user, execute the following:

 

$ ac -d sanjay

Jul 2 total 12.85

Aug 25 total 5.05

Sep 3 total 1.03

Sep 4 total 5.37

Dec 24 total 8.15

Dec 29 total 1.42

Today total 2.95

Hack #18 -> Diff Command

Diff command compares two different files and reports the difference.

The output of the diff command is very cryptic and not straight forward

to read.

Syntax: diff [options] file1 file2

 

What was modified in my new file when compare to my old

file?

 

The option -w in the diff command will ignore the white space while

performing the comparison.

In the following diff output:

  • The lines above —, indicates the changes happened in first file in

the diff command (i.e name_list.txt).

  • The lines below —, indicates the changes happened to the

second file in the diff command (i.e name_list_new.txt). The lines

that belong to the first file starts with < and the lines of second

file starts with >.

 

# diff -w name_list.txt name_list_new.txt

2c2,3

< John Doe

—

> John M Doe

> Jason Bourne

Hack #17 -> Stat Command

Stat command can be used either to check the status/properties of a

single file or the filesystem.

Display statistics of a file or directory.

 

$ stat /etc/my.cnf

File: `/etc/my.cnf’

Size: 346 Blocks: 16 IO Block: 4096 regular file

Device: 801h/2049d Inode: 279856 Links: 1

Access: (0644/-rw-r–r–) Uid: (0/root) Gid: (0/root)

Access: 2009-01-01 02:58:30.000000000 -0800

Modify: 2006-06-01 20:42:27.000000000 -0700

Change: 2007-02-02 14:17:27.000000000 -0800

$ stat /home/ramesh

File: `/home/ramesh’

Size: 4096 Blocks: 8 IO Block: 4096

directory

Device: 803h/2051d Inode: 5521409 Links: 7

Access: (0755/drwxr-xr-x) Uid: (401/ramesh) Gid:

(401/ramesh)

Access: 2009-01-01 12:17:42.000000000 -0800

Modify: 2009-01-01 12:07:33.000000000 -0800

Change: 2009-01-09 12:07:33.000000000 -0800

 

Display the status of the filesystem using option –f

 

$ stat -f /

File: “/”

ID: 0 Namelen: 255 Type: ext2/ext3

Blocks: Total: 2579457 Free: 2008027 Available:

1876998 Size: 4096

Inodes: Total: 1310720 Free: 1215892

%d bloggers like this: