Why you don't read lines with “for” in bash?
I've read that using "for" is not a good way to read lines in bash.
Many people says this is clumsy and inefficient at best, and fails in many cases.
I'd like to know what is the best way to read lines in bash. Thanks.
For example:
$ for i in $(<afile); do echo "$i"; done
bash readline
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
add a comment |
I've read that using "for" is not a good way to read lines in bash.
Many people says this is clumsy and inefficient at best, and fails in many cases.
I'd like to know what is the best way to read lines in bash. Thanks.
For example:
$ for i in $(<afile); do echo "$i"; done
bash readline
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
1
Do you have an example of a bash script you've written using "for"?
– L2G
Dec 1 '11 at 17:39
for example: $ for i in $(<afile); do echo "$i"; done
– Kyrol
Dec 1 '11 at 17:51
add a comment |
I've read that using "for" is not a good way to read lines in bash.
Many people says this is clumsy and inefficient at best, and fails in many cases.
I'd like to know what is the best way to read lines in bash. Thanks.
For example:
$ for i in $(<afile); do echo "$i"; done
bash readline
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
I've read that using "for" is not a good way to read lines in bash.
Many people says this is clumsy and inefficient at best, and fails in many cases.
I'd like to know what is the best way to read lines in bash. Thanks.
For example:
$ for i in $(<afile); do echo "$i"; done
bash readline
bash readline
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
edited Mar 7 '13 at 5:59
Karan
49.4k1489161
49.4k1489161
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
asked Dec 1 '11 at 17:30
KyrolKyrol
52951533
52951533
1
Do you have an example of a bash script you've written using "for"?
– L2G
Dec 1 '11 at 17:39
for example: $ for i in $(<afile); do echo "$i"; done
– Kyrol
Dec 1 '11 at 17:51
add a comment |
1
Do you have an example of a bash script you've written using "for"?
– L2G
Dec 1 '11 at 17:39
for example: $ for i in $(<afile); do echo "$i"; done
– Kyrol
Dec 1 '11 at 17:51
1
1
Do you have an example of a bash script you've written using "for"?
– L2G
Dec 1 '11 at 17:39
Do you have an example of a bash script you've written using "for"?
– L2G
Dec 1 '11 at 17:39
for example: $ for i in $(<afile); do echo "$i"; done
– Kyrol
Dec 1 '11 at 17:51
for example: $ for i in $(<afile); do echo "$i"; done
– Kyrol
Dec 1 '11 at 17:51
add a comment |
1 Answer
1
active
oldest
votes
You are talking about for line in $(cat file); do and similar constructs. This is:
- Inefficient because bash has to spawn a subshell and execute
cat, readcat's output – the entire file – into memory, parse it (which is the slowest part), only then iterate over all data - Unreliable because bash performs word-splitting on the data – not only does it split on newline characters, but also on anything in $IFS (spaces, tabs...)
(If you use $(<...) instead of $(cat ...), you save two milliseconds on Linux, but all other downsides remain.)
A much better option is to use read in a while loop:
while read -r line; do
echo "Input: $line"
done < myfile.txt
Or from a program:
cat file1 file2 file3 |
while read -r line; do
echo "Input: $line"
done
This only reads as much as is needed, does not perform unnecessary processing but allows custom field splitting, and is many times faster and less resource-demanding on large files.
If you're trying to work with the output of find, you should use the same pattern:
find . -name "foo" -type f -print0 | while read -r -d '' file; do
echo "File: $file"
done
See also:
- Greg's Wiki. Why you don't read lines with "for"
- Greg's Wiki. Bash FAQ #001: How can I read a file line-by-line?
edited Feb 19 at 9:12
slhck
162k47448471
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f363559%2fwhy-you-dont-read-lines-with-for-in-bash%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You are talking about for line in $(cat file); do and similar constructs. This is:
- Inefficient because bash has to spawn a subshell and execute
cat, readcat's output – the entire file – into memory, parse it (which is the slowest part), only then iterate over all data - Unreliable because bash performs word-splitting on the data – not only does it split on newline characters, but also on anything in $IFS (spaces, tabs...)
(If you use $(<...) instead of $(cat ...), you save two milliseconds on Linux, but all other downsides remain.)
A much better option is to use read in a while loop:
while read -r line; do
echo "Input: $line"
done < myfile.txt
Or from a program:
cat file1 file2 file3 |
while read -r line; do
echo "Input: $line"
done
This only reads as much as is needed, does not perform unnecessary processing but allows custom field splitting, and is many times faster and less resource-demanding on large files.
If you're trying to work with the output of find, you should use the same pattern:
find . -name "foo" -type f -print0 | while read -r -d '' file; do
echo "File: $file"
done
See also:
- Greg's Wiki. Why you don't read lines with "for"
- Greg's Wiki. Bash FAQ #001: How can I read a file line-by-line?
edited Feb 19 at 9:12
slhck
162k47448471
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
add a comment |
You are talking about for line in $(cat file); do and similar constructs. This is:
- Inefficient because bash has to spawn a subshell and execute
cat, readcat's output – the entire file – into memory, parse it (which is the slowest part), only then iterate over all data - Unreliable because bash performs word-splitting on the data – not only does it split on newline characters, but also on anything in $IFS (spaces, tabs...)
(If you use $(<...) instead of $(cat ...), you save two milliseconds on Linux, but all other downsides remain.)
A much better option is to use read in a while loop:
while read -r line; do
echo "Input: $line"
done < myfile.txt
Or from a program:
cat file1 file2 file3 |
while read -r line; do
echo "Input: $line"
done
This only reads as much as is needed, does not perform unnecessary processing but allows custom field splitting, and is many times faster and less resource-demanding on large files.
If you're trying to work with the output of find, you should use the same pattern:
find . -name "foo" -type f -print0 | while read -r -d '' file; do
echo "File: $file"
done
See also:
- Greg's Wiki. Why you don't read lines with "for"
- Greg's Wiki. Bash FAQ #001: How can I read a file line-by-line?
edited Feb 19 at 9:12
slhck
162k47448471
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
add a comment |
You are talking about for line in $(cat file); do and similar constructs. This is:
- Inefficient because bash has to spawn a subshell and execute
cat, readcat's output – the entire file – into memory, parse it (which is the slowest part), only then iterate over all data - Unreliable because bash performs word-splitting on the data – not only does it split on newline characters, but also on anything in $IFS (spaces, tabs...)
(If you use $(<...) instead of $(cat ...), you save two milliseconds on Linux, but all other downsides remain.)
A much better option is to use read in a while loop:
while read -r line; do
echo "Input: $line"
done < myfile.txt
Or from a program:
cat file1 file2 file3 |
while read -r line; do
echo "Input: $line"
done
This only reads as much as is needed, does not perform unnecessary processing but allows custom field splitting, and is many times faster and less resource-demanding on large files.
If you're trying to work with the output of find, you should use the same pattern:
find . -name "foo" -type f -print0 | while read -r -d '' file; do
echo "File: $file"
done
See also:
- Greg's Wiki. Why you don't read lines with "for"
- Greg's Wiki. Bash FAQ #001: How can I read a file line-by-line?
edited Feb 19 at 9:12
slhck
162k47448471
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
You are talking about for line in $(cat file); do and similar constructs. This is:
- Inefficient because bash has to spawn a subshell and execute
cat, readcat's output – the entire file – into memory, parse it (which is the slowest part), only then iterate over all data - Unreliable because bash performs word-splitting on the data – not only does it split on newline characters, but also on anything in $IFS (spaces, tabs...)
(If you use $(<...) instead of $(cat ...), you save two milliseconds on Linux, but all other downsides remain.)
A much better option is to use read in a while loop:
while read -r line; do
echo "Input: $line"
done < myfile.txt
Or from a program:
cat file1 file2 file3 |
while read -r line; do
echo "Input: $line"
done
This only reads as much as is needed, does not perform unnecessary processing but allows custom field splitting, and is many times faster and less resource-demanding on large files.
If you're trying to work with the output of find, you should use the same pattern:
find . -name "foo" -type f -print0 | while read -r -d '' file; do
echo "File: $file"
done
See also:
- Greg's Wiki. Why you don't read lines with "for"
- Greg's Wiki. Bash FAQ #001: How can I read a file line-by-line?
edited Feb 19 at 9:12
slhck
162k47448471
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
edited Feb 19 at 9:12
slhck
162k47448471
edited Feb 19 at 9:12
slhck
162k47448471
edited Feb 19 at 9:12
slhck
162k47448471
162k47448471
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
answered Dec 1 '11 at 17:45
grawitygrawity
241k37510566
241k37510566
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
add a comment |
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
Very nice answer! You showed me what I wanted understand! thanks!
– Kyrol
Dec 2 '11 at 13:04
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f363559%2fwhy-you-dont-read-lines-with-for-in-bash%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Do you have an example of a bash script you've written using "for"?
– L2G
Dec 1 '11 at 17:39
for example: $ for i in $(<afile); do echo "$i"; done
– Kyrol
Dec 1 '11 at 17:51