Linux command line remove lines from file

Howto: Linux command line utilities for removing blank lines from text files

Q. I want to change the formatting of a file. I just wanted to remove all blank lines from text file. How do I achieve this task w/o spending much time?

A. Yes, you do not have to waste your time making manual changes to files. Both Linux and UNIX systems come with file manipulation tools that can be used to remove all blank lines very quickly.

Task: Remove blank lines using sed

Type the following command:
$ sed ‘/^$/d’ input.txt > output.txt

Task: Remove blank lines using grep

$ grep -v ‘^$’ input.txt > output.txt

Both grep and sed use special pattern ^$ that matchs the blank lines. Grep -v option means print all lines except blank line.

Let us say directory /home/me/data/*.txt has all text file. Use following for loop (shell script) to remove all blank lines from all files stored in /home/me/data directory:

• No ads and tracking
• In-depth guides for developers and sysadmins at Opensourceflare✨
• Join my Patreon to support independent content creators and start reading latest guides:
  • How to set up Redis sentinel cluster on Ubuntu or Debian Linux
  • How To Set Up SSH Keys With YubiKey as two-factor authentication (U2F/FIDO2)
  • How to set up Mariadb Galera cluster on Ubuntu or Debian Linux
  • A podman tutorial for beginners – part I (run Linux containers without Docker and in daemonless mode)
  • How to protect Linux against rogue USB devices using USBGuard

Join Patreon ➔

Updated for accuracy.

🐧 Get the latest tutorials on Linux, Open Source & DevOps via

Category	List of Unix and Linux commands
Documentation	help • mandb • man • pinfo
Disk space analyzers	df • duf • ncdu • pydf
File Management	cat • cp • less • mkdir • more • tree
Firewall	Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04
Linux Desktop Apps	Skype • Spotify • VLC 3
Modern utilities	bat • exa
Network Utilities	NetHogs • dig • host • ip • nmap
OpenVPN	CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04
Package Manager	apk • apt
Processes Management	bg • chroot • cron • disown • fg • glances • gtop • jobs • killall • kill • pidof • pstree • pwdx • time • vtop
Searching	ag • grep • whereis • which
Shell builtins	compgen • echo • printf
Text processing	cut • rev
User Information	groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w
WireGuard VPN	Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04

Comments on this entry are closed.

Is that last shell script wrong? The for loop is using variable f, and inside the loop, everything is using variable i. I think the script should look more like:

#!/bin/sh
files=”/home/me/data/*.txt”
# Next line is changed to use variable i, not f
for i in $files
do
sed ‘/^$/d’ $i > $i.out
mv $i.out $i
done

Little fix in this BASH script:

#!/bin/bash
#!/bin/sh
files=”/home/tiger/test_moduls/*.txt”
for i in $files
do
sed ‘/^ *$/d’ $i > $i.out
mv $i.out $i
done

Thanks for the heads up. The faq has been updated.

When writing to the same file, or a different file and using the redirect > the file ends up blank.

This removes blank lines from input.txt

sed ‘/^$/d’ -i input.txt

I tried to use the command sed given above to remove blank line from my file.

The file looks good except one strange behaviour:
“Always” the last line of the file is removed..for example:
test.p is the file and following are the contents of the file:

BEFORE RUNNING THE SED COMMAND:
“this is test 1.

AFTER RUNNING THE SED COMMAND:
“this is test 1.
this is test 2.”

As you can see the last line (“this is test 3.”) is removed after running the command.

If I put the blank line at the end it removes the blank line i.e.
“this is test 1.

becomes…….
“this is test 1.
this is test 2.
this is test 3.”

Do let me know what I should do in this case?

Waiting for your prompt reply…!

Thanks and Best Regards,
Prashant Deshani

Actually, grep’s -v (–invert-match) option inverts the sense of matching, thus selecting non-matching lines.
Therefore, only when combining it with the pattern ‘^$’ does it mean to select everything except blank lines.
Just thought the way you explained it might be confusing for beginners. When I was new to shell and Linux, most of the guides or forums were useless because they all assumed that the person that needs help has some high understanding of shell or Linux already, and thus they use a lot of terms that person would have never heard of before or they don’t explain anything. Once you have a basic understanding, the learning comes easier, but in the beginning, the learning curve has an almost horizontal slope.

Hi,
I am Siva Saran
I am new to linux environment
can any on tell plz that how to get a particular line from a file using linux commands
For example say that inputfile has lines as follows
total 0
d [RWCEAFMS] MRohit 512 Apr 15 03:15 3.1_SP1_Beta1_127
d [RWCEAFMS] MRohit 512 Apr 14 03:26 3.1_SP1_125 d [RWCEAFMS] MRohit 512 Apr 14 03:26 3.1_SP1_259
d [RWCEAFMS] MRohit 512 Apr 14 03:26 3.0_SP4_IR3_62

Now I want the line which is having SP1_125 ie., line number 3 and store it in an variable
plz can any tell the answer plz
can u mail to

what if there’s spaces and tab in the line I want to delete?

This is old, but in case no-one told you:

\w is the special regular expression for whitespace. Both the sed and grep matches should be changed to:

Using the command
grep -R “SP1_125”
you can get the lines with SP1_125.
if you want to assign to a variable then use
var_a=$(grep -R “SP1_125” )

where is the name of file

var_a = $(grep -R “SP1_125” )

Hi,
var_a=$(grep -R “SP1_125” filename)

This is quite helpful. Thanks!!

how count all lines in all files in current dir and omit empty lines with wc, grep, cut and bc commands

echo `wc -l * | grep total | cut -f2 -d’ ‘` – `grep -in “^$” * | wc -l ` | bc

@sobi3ch
LOL, wut? Wanna win an obfuscation contest?

would be better, because it doesn’t count lines with only whitespaces, and uses only two instead of seven(!) programs.

will print you the number of lines, excluding empty lines and lines starting with “#” for every file separately in the current directory.

All you are trying to do is this:

The example should read

Because
a) using “^\w*$” instead of “^*$” removes also lines consisting only of whitespaces
b) sed Walter Apr 16, 2011 @ 19:35

“sed Walter Apr 16, 2011 @ 20:14

“sed pj Jul 13, 2011 @ 6:44

I would like to know how i can write a shell script to delete “datetime message” and a particular patern exists? in short I want my file to this following output.

### original file ###
datetime message
2011-07-13 13:45:35 Hello World of War Craft
2011-07-13 13:48:43 This is a Test of text

### output file ###
20110713134535 | Hello World of War Craft
20110713134843 | This is a Test of text

You need to process the file using while loop:

There may be a better solution to reduced awk, but I’m too lazy to try it out 😉

Thanks for fast reply,

when I run this shell script
# while IFS= read -r line; do echo “$line” | sed -e ‘s/-//g’ -e ‘s/://g’ | awk ‘< print $1$2 ” | ” substr($0, index($0,$3))>’; done testdb1.pl;

the file output is below: the line 1 not remove and added datetimemessage, is there any way to remove the line 1 which is “datetimemessage | datetime message”

datetimemessage | datetime message
20110713134535 | Hello World of War Craft
20110713134843 | This is a Test of text

while IFS= read -r line; do echo “$line” | sed -e ‘s/-//g’ -e ‘s/://g’ | awk ‘< print $1$2 ” | ” substr($0, index($0,$3))>’; done testdb1.pl;

correction
‘; done testdb.pl “>testdb.pl”

I don’t if is the right to post this issue,
is there any way to dump the new insert data to a textfile with shell script, for ie. after insertion to the mysql database execute the shell script to append and append the new record?

Is there a way to add an empty line for every 5 lines of data using grep?

line1
line2
line3
line4
line5
(empty line)
line6
line7
line8
line9
line10
(empty line)

Источник

Your Own Linux.

Linux How To’s | Bash Scripting | Python

Monday, 20 April 2015

Sed Command in Linux — Delete Lines from a File

This is the third article of the «Super sed ‘ Series», in which we will learn how to delete lines from a file using line numbers and regular expressions. In the first two articles, we have learned, how to print lines using sed and how to append and insert lines in a file using sed. If you are familiar with sed print syntaxes, then this article would be pretty easy to understand.

For those learners who are new to sed , let’s have a brief introduction of the Super sed :

• sed stand for Stream EDitor and it being based on the ed editor, it borrows most of the commands from the ed . It was developed by Lee E. McMahon of Bell Labs.
• sed offers large range of text transformations that include printing lines, deleting lines, editing line in-place, search and replace, appending and inserting lines, etc.
• sed is useful whenever you need to perform common editing operations on multiple lines without using ‘vi’ editor.
• Whenever sed is executed on an input file or on the contents from stdin, sed reads the file line-by-line and after removing the trailing newline, places it in the «Pattern space», where the commands are executed on them after conditions (as in case of regex matching) are verified, and then printed on the stdout.

Deleting Lines from a File using sed

Before we start, just remember two points:

1. sed «d» command lets us print specific lines based on the line number or regex provided.
2. When ^ means beginning of the line and $ denotes end of the line, ^$ makes a «Blank Line», very useful while removing empty lines from a file.

For our better understanding, let us have a file sedtest.txt with contents as follows:

A. sed — Delete Lines with Line Number

1. Delete ‘N’th line

This will remove ‘N’th line in the FILE.txt .

Example:
To delete 1st line,

While, to remove last line,

If you run above commands and inspect the file sedtest.txt , you would find that, the original contents of that file would not change. In case you want to remove lines in the file and save the changes (i.e. edit the file in place), you will have to use the option -i as shown in below example:

2. Delete all Lines starting from ‘M’th up to ‘N’th

This will remove the block of lines starting at line number M and ending at line number N .

Example:
To delete block of lines from 3rd line to 8th line.

Similarly, in order to delete lines starting from 5th up to last line, you would run-

3. Delete Every ‘M’th Line Starting from ‘N’th Line

This will start from Nth line and delete every Mth line coming after that. Note that, Nth line will also be deleted.

Example:
To delete every alternate line staring from 2nd one.

B. sed — Delete Lines using Regular Expression/Pattern

1. Delete lines containing a specific Pattern

This will delete all those lines that contain the pattern provided.

2. Delete all those lines not containing the Pattern

This will delete all those lines which do not include the pattern provided.

3. Delete block of lines starting from pattern matching line

This will remove the lines from the one where pattern matches, till ‘N’th line.

Similarly, so as to delete all the lines starting from pattern matching line till the end, you would use ‘/PATTERN/,$ d’ as follows —

4. Delete block of lines starting from Nth and ending at pattern matching line

This will delete the lines starting from the ‘N’th line, till the one where pattern matches.

5. Delete block of lines between two Pattern matches

This will start deleting lines from 1st pattern match till 2nd pattern match.

6. Deleting all blank lines

7. Deleting lines containing either of two patterns

That was all about the third article on sed command. More articles on sed are coming soon. So, stay tuned. Of course, do not forget to share your feedback in the comment section below.

Источник