sed Command in Linux (Find and Replace Strings in Files)

Imagine needing to change a specific configuration setting across dozens of files, or update a repeated piece of text throughout a large log file. Manually editing each instance would be incredibly tedious. This is precisely where the powerful Linux sed command (Stream EDitor) excels. More than just a simple find-and-replace tool, the sed command is a versatile utility designed to automate text transformations on the fly, allowing you to perform complex edits with concise instructions and saving countless hours.

This guide will walk you through everything you need to know about sed, from the basics of finding and replacing text to more advanced tricks like working with regular expressions and editing multiple files at once. By the end, you’ll have the confidence to use sed effectively and avoid common pitfalls. Let’s get started and see how sed can make your workflow smoother.

Understanding the sed Command

Before jumping into examples, it helps to get familiar with the basics of the sed command. On most Linux distributions, GNU sed comes pre-installed, so you’re ready to go. For the most comprehensive details, you can always refer to the official GNU sed manual. At its core, sed is about making text manipulation simple and efficient.

Here’s the general syntax for finding and replacing text:

sed -i 's/SEARCH_REGEX/REPLACEMENT/g' INPUTFILE

Let’s break it down step by step:

• -i: This tells sed to edit the file in place. If you want to keep a backup of the original file, you can add an extension like .bak (e.g., -i.bak). On macOS (BSD sed), if you don\’t want a backup, you must provide an empty string as an argument to -i, for example, -i ''. GNU sed allows -i without an extension if no backup is desired.
• s: The substitute command, used for search-and-replace operations.
• Delimiters (/): While / is the default delimiter, you can use others like | or : if it makes your command easier to read.
• SEARCH_REGEX: The string or regular expression pattern you’re looking for.
• REPLACEMENT: The text you want to replace it with.
• g: The global flag. Without it, only the first match on each line is replaced. Add this to replace all matches.

With this foundation, you’re ready to start using sed to tackle text editing tasks of all sizes.

Basic sed Command Usage for String Replacement

Let’s start with the basics of the sed command by looking at how to find and replace text in a file. Here’s an example using a file called file.txt with the following content:

123 Foo foo foo
foo /bin/bash Ubuntu foobar 456

Replace the First Instance of a String

To replace only the first occurrence of a string on each line, you can use the following command:

sed -i 's/foo/linux/' file.txt

Output:

123 Foo linux foo
linux /bin/bash Ubuntu foobar 456

Notice that only the first foo on each line was replaced, while other instances remain unchanged.

Replace All Occurrences of a String

To replace every occurrence of a string on each line, add the g flag for “global”:

sed -i 's/foo/linux/g' file.txt

Output:

123 Foo linux linux
linux /bin/bash Ubuntu linuxbar 456

This replaces every instance of foo, including when it appears as part of another word, such as foobar.

Avoid Matching Substrings

If you don’t want sed to replace partial matches (like foo inside foobar), you can use word boundaries. In GNU sed, you can use \b as follows:

sed -i 's/\bfoo\b/linux/g' file.txt

Note: The \b word boundary is supported in GNU sed. On macOS (BSD sed), use [[:<:]]foo[[:>:]] instead:

sed -E 's/[[:<:]]foo[[:>:]]/linux/g' file.txt

Output:

123 Foo linux linux
linux /bin/bash Ubuntu foobar 456

Case-Insensitive Replacement

To make the replacement case-insensitive, include the I flag (GNU sed only):

sed -i 's/foo/linux/gI' file.txt

Note: The I flag is available in GNU sed. On macOS (BSD sed), use a character class for case-insensitivity, e.g., [Ff]oo. Remember to use -i '' with BSD sed if you are editing in-place without a backup, as shown below:

sed -i '' 's/[Ff]oo/linux/g' file.txt

Output:

123 linux linux linux
linux /bin/bash Ubuntu linuxbar 456

Replace Strings Containing Special Characters

If the string you’re replacing includes special characters like /, you’ll need to escape them with a backslash (\). For example, to replace /bin/bash with /usr/bin/zsh:

sed -i 's/\/bin\/bash/\/usr\/bin\/zsh/g' file.txt

Or, use a different delimiter (like |) to make the command more readable. In fact, you can use almost any character as a delimiter, such as #, :, or _, as long as it does not appear in your search pattern or replacement string:

sed -i 's|/bin/bash|/usr/bin/zsh|g' file.txt

Output:

123 Foo foo foo
foo /usr/bin/zsh Ubuntu foobar 456

Quick Tip:

• Test Before Committing Changes: Run your sed commands without the -i option to preview the changes before modifying the file.
• Always Create Backups: Use -i.bak to save a backup of the original file, just in case. For example:

sed -i.bak 's/foo/linux/g' file.txt

What’s Next?

If you’re ready to take your skills to the next level, the next section will cover advanced sed command features. This includes working with regular expressions, line-specific replacements, and handling larger files or directories.

Advanced sed Command Operations for Strings

Once you’ve mastered the basics of sed, you can move on to its more advanced capabilities. With support for regular expressions, line-specific operations, and complex text transformations, the sed command becomes an even more powerful tool for handling even the most intricate text manipulation tasks.

Using Regular Expressions with the sed Command for Advanced Matching

Regular expressions allow you to match patterns, not just literal strings when using the sed command. For instance, to replace all 3-digit numbers with the word number:

sed -i 's/\b[0-9]\{3\}\b/number/g' file.txt

Output:

number Foo foo foo
foo /bin/bash Ubuntu foobar number

This matches only 3-digit numbers (e.g., 123 or 456) and replaces them, leaving other numbers unaffected.

Reusing the Matched Pattern with &

The & character corresponds to the matched pattern and can be used in the replacement string. For example, if you want to surround every 3-digit number with curly braces:

sed -i 's/\b[0-9]\{3\}\b/{&}/g' file.txt

Output:

{123} Foo foo foo
foo /bin/bash Ubuntu foobar {456}

Here, {&} inserts the matched number between the curly braces.

Line-Specific String Replacements using the sed Command

You can target specific lines or ranges of lines for replacement with the sed command:

• Replace text on a specific line (e.g., line 2):

sed '2s/foo/linux/' file.txt

• Replace text in a range of lines (e.g., lines 2 to 4):

sed '2,4s/foo/linux/' file.txt

The sed command gives you fine-grained control over where replacements occur, avoiding unintended changes.

Deleting Lines Matching a Pattern

You can delete all lines that contain a specific string using a simple sed command. For example, to remove any line with the word Ubuntu:

sed '/Ubuntu/d' file.txt

Output:

123 Foo foo foo

Inserting or Appending Lines

To add a line before or after a specific pattern, use the i (insert) or a (append) commands. For example:

• Insert a line before every line containing foo:

Note: On macOS (BSD sed), the syntax for inserting/appending lines requires a backslash before the newline, and the text to be inserted/appended must appear on the immediately following line. For example, to insert:

sed '/foo/i\
This is a new line' file.txt

And to append:

sed '/foo/a\
This is a new line' file.txt

Ensure there are no trailing spaces after the backslash.

Changing Entire Lines

To replace entire lines that match a pattern, use the c (change) command. For example, to replace all lines containing foo with This is a new line:

sed '/foo/c\
This is a new line' file.txt

Note: On macOS (BSD sed), the syntax for changing lines is similar to inserting/appending and requires a backslash before the newline, with the replacement text on the next line. For example:

sed '/foo/c\
This is a new line' file.txt

Output (assuming both original lines contained ‘foo’):

This is a new line
This is a new line

Combining Multiple sed Command Instructions

You can chain multiple sed command instructions by separating them with semicolons. For instance, to replace foo with linux, delete lines containing Ubuntu, and append a line after bar:

sed -i 's/foo/linux/g;/Ubuntu/d;/bar/a\
This line was added' file.txt

This approach lets you perform complex transformations in a single command.

Using the sed Command with Large Files or Multiple Directories

When working with large files or directories, combining the sed command with other tools like find or grep becomes incredibly useful:

• Recursive Search and Replace: Replace all occurrences of foo with bar in the current directory and its subdirectories:

find . -type f -exec sed -i 's/foo/bar/g' {} +

Warning: This sed command, when combined with find -exec, will edit all files, including potentially binaries, in the directory tree. To avoid unintended changes, it is highly recommended to perform a dry run first. You can do this by replacing -exec sed -i 's/foo/bar/g' {} + with -print to list files that would be affected, or by running the sed command without -i and redirecting output to inspect. Always restrict to specific text file types if possible (see next example) or use grep to filter files first.

• Target Specific File Types: Replace foo with bar only in .txt files:

find . -type f -name "*.txt" -exec sed -i 's/foo/bar/g' {} +

Using grep for Matching Files: Search for files containing foo and replace it with bar, while creating backups:

grep -rlZ 'foo' . | xargs -0 sed -i.bak 's/foo/bar/g'

These techniques allow you to handle large-scale text manipulations with precision.

Advanced Tip: Speed Up sed with Unbuffered Mode

When working with large files, enable unbuffered mode using the -u option. This processes the file in chunks instead of loading it all into memory, improving performance:

sed -u 's/foo/bar/g' largefile.txt

Note: The -u (unbuffered) option is available in GNU sed and is rarely needed for typical text processing. It may not be available or necessary on macOS (BSD sed).

Troubleshooting Common sed Command Issues

While the sed command is powerful, you might encounter a few common issues. Here’s how to troubleshoot them:

1. Changes Not Happening or Incorrect Replacements with the sed command

• Forgetting Global Replacement (g): If only the first match on each line is replaced, ensure you’ve added the g flag to your substitute command (e.g., s/old/new/g) to replace all occurrences.
• Regex Errors: Special characters in your search pattern (like ., *, [, \) need to be escaped with a backslash (\). For example, to match a literal dot, use \.. Alternatively, for paths with slashes, use a different delimiter: sed 's|/path/to/old|/path/to/new|g'.
• Word Boundaries: If sed replaces parts of words you didn’t intend (e.g., “man” in “manual”), use word boundaries. For GNU sed, use \b (e.g., s/\bman\b/person/g). For BSD sed (macOS), use [[:<:]] and [[:>:]] (e.g., s/[[:<:]]man[[:>:]]/person/g).
• Case Sensitivity: By default, sed is case-sensitive. For case-insensitive search in GNU sed, use the I flag (e.g., s/text/replacement/gI). For BSD sed, you might need to use character classes like [Tt][Ee][Xx][Tt].

2. Differences Between GNU sed (Linux) and BSD sed (macOS) when using the sed command

Syntax and feature differences between GNU sed (common on Linux) and BSD sed (used on macOS) are a frequent source of confusion when working with the sed command:

• In-place Editing (-i):
  • GNU sed: -i edits in place without a backup. -i.bak creates a backup. Note: GNU sed creates the backup file specified with -i.SUFFIX even if no actual changes are made to the file content.
  • BSD sed: -i requires an extension for backup (e.g., -i '.bak'). For no backup, provide an empty string: -i ''.
• Extended Regular Expressions:
  • GNU sed: Supports both -r and -E. The POSIX standard is -E, which is recommended for better portability.
  • BSD sed: Uses -E.
• Insert (i), Append (a), Change (c) Commands:
  • BSD sed: These commands strictly require the text to be on a new line, preceded by a backslash. Example: sed '/pattern/a\ Appended text' file.txt.
  • GNU sed: More flexible, but following the BSD syntax is generally safer for portability.

Tip: Always check your system’s man sed page for specifics.

3. “sed: -i may not be used with stdin” Error for the sed command

The -i option (for in-place editing with the sed command) modifies files directly and cannot be used when sed receives input from a pipe (stdin). If you’re piping output from another command to sed and want to save the changes, redirect the output to a new file:

some_command | sed 's/old/new/g' > output_file.txt

If you need to modify the original file in such a scenario, use a temporary file or more advanced shell constructs.

4. Avoiding Accidental File Modification

Before using -i to modify a file directly, it’s crucial to:

• Test without -i: Run your sed command without -i first. This prints the modified output to the terminal, allowing you to verify the changes are correct before applying them to the file.
• Create Backups: Use -i.bak (GNU sed) or -i '.bak' (BSD sed) to create a backup of the original file. If something goes wrong, you can easily restore it.

5. Using sed on Binary Files

sed is designed for text files. Using it on binary files (like images or executables) can corrupt them. For binary file manipulation, tools like xxd or hexedit are more appropriate.

Conclusion: Mastering sed for Everyday Text Editing on Linux

The sed command is a versatile Linux utility for text editing, useful for tasks ranging from basic substitutions to complex file transformations. Learning to use sed effectively can save considerable time and improve your ability to manipulate text, a key skill for developers, system administrators, and Linux users.

This guide covered the fundamentals and some advanced uses of sed. Incorporating sed into your regular command-line usage will allow you to tailor it to your needs, whether for configuration management, log processing, or data refinement.

Share Your Thoughts & Experiences

We’re interested in your experiences with the sed command. Have you used it in your projects? Feel free to share any challenges, tips, or tricks you’ve discovered in the comments below. Your insights are helpful to other Linux users and for improving content like this.