How to create a Loop in Shell Script to do a specific task?
I am trying to clean a Huge database of emails, but when I use the following Grep command, I get "Memory Exhausted" Error.
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' EMAILS.txt) > RESULT.txt
The data set in both file are really huge like more than 5 Million emails, therefore the Linux server gives me "Memory Exhausted" Error.
Therefore I decided to split the file in into small files with 10,000 emails each to process.
split -d -l 10000 EMAILS.txt Segment
How to create a Bash Script with a loop, where it checks for the created split file in increasing order and starts processing from the first file created - Eg: Segment00
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment00) > RESULT.txt
...then automatically loop the same command and substitute the second segment file in the command - Eg: Segment01
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment01) > RESULT.txt
...carry the loop until the last segment and then finally terminate.
Can you please help me write such Bash script? I cannot figure out how to automatically substitute the segmented files in increasing order automatically in the loop to write a shell script.
Please help.
linux command-line bash shell loopback
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
add a comment |
I am trying to clean a Huge database of emails, but when I use the following Grep command, I get "Memory Exhausted" Error.
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' EMAILS.txt) > RESULT.txt
The data set in both file are really huge like more than 5 Million emails, therefore the Linux server gives me "Memory Exhausted" Error.
Therefore I decided to split the file in into small files with 10,000 emails each to process.
split -d -l 10000 EMAILS.txt Segment
How to create a Bash Script with a loop, where it checks for the created split file in increasing order and starts processing from the first file created - Eg: Segment00
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment00) > RESULT.txt
...then automatically loop the same command and substitute the second segment file in the command - Eg: Segment01
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment01) > RESULT.txt
...carry the loop until the last segment and then finally terminate.
Can you please help me write such Bash script? I cannot figure out how to automatically substitute the segmented files in increasing order automatically in the loop to write a shell script.
Please help.
linux command-line bash shell loopback
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
add a comment |
I am trying to clean a Huge database of emails, but when I use the following Grep command, I get "Memory Exhausted" Error.
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' EMAILS.txt) > RESULT.txt
The data set in both file are really huge like more than 5 Million emails, therefore the Linux server gives me "Memory Exhausted" Error.
Therefore I decided to split the file in into small files with 10,000 emails each to process.
split -d -l 10000 EMAILS.txt Segment
How to create a Bash Script with a loop, where it checks for the created split file in increasing order and starts processing from the first file created - Eg: Segment00
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment00) > RESULT.txt
...then automatically loop the same command and substitute the second segment file in the command - Eg: Segment01
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment01) > RESULT.txt
...carry the loop until the last segment and then finally terminate.
Can you please help me write such Bash script? I cannot figure out how to automatically substitute the segmented files in increasing order automatically in the loop to write a shell script.
Please help.
linux command-line bash shell loopback
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
I am trying to clean a Huge database of emails, but when I use the following Grep command, I get "Memory Exhausted" Error.
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' EMAILS.txt) > RESULT.txt
The data set in both file are really huge like more than 5 Million emails, therefore the Linux server gives me "Memory Exhausted" Error.
Therefore I decided to split the file in into small files with 10,000 emails each to process.
split -d -l 10000 EMAILS.txt Segment
How to create a Bash Script with a loop, where it checks for the created split file in increasing order and starts processing from the first file created - Eg: Segment00
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment00) > RESULT.txt
...then automatically loop the same command and substitute the second segment file in the command - Eg: Segment01
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment01) > RESULT.txt
...carry the loop until the last segment and then finally terminate.
Can you please help me write such Bash script? I cannot figure out how to automatically substitute the segmented files in increasing order automatically in the loop to write a shell script.
Please help.
linux command-line bash shell loopback
linux command-line bash shell loopback
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
asked Feb 8 at 6:46
Joney WalkerJoney Walker
235
235
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You can use for example for loop like this:
for i in Segment??
do
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
done
This will exec the command with all Segment<symbol><symbol> files and ADD result to RESULT.txt
Based on the comment the command should be:
pv "RESULT-1.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
1
Right. I use such way to restrict the filenames. You can use alsoSegment*if you are sure only files in question start with this string
– Romeo Ninov
Feb 8 at 7:13
1
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
1
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
|
show 8 more comments
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1403418%2fhow-to-create-a-loop-in-shell-script-to-do-a-specific-task%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can use for example for loop like this:
for i in Segment??
do
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
done
This will exec the command with all Segment<symbol><symbol> files and ADD result to RESULT.txt
Based on the comment the command should be:
pv "RESULT-1.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
1
Right. I use such way to restrict the filenames. You can use alsoSegment*if you are sure only files in question start with this string
– Romeo Ninov
Feb 8 at 7:13
1
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
1
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
|
show 8 more comments
You can use for example for loop like this:
for i in Segment??
do
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
done
This will exec the command with all Segment<symbol><symbol> files and ADD result to RESULT.txt
Based on the comment the command should be:
pv "RESULT-1.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
1
Right. I use such way to restrict the filenames. You can use alsoSegment*if you are sure only files in question start with this string
– Romeo Ninov
Feb 8 at 7:13
1
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
1
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
|
show 8 more comments
You can use for example for loop like this:
for i in Segment??
do
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
done
This will exec the command with all Segment<symbol><symbol> files and ADD result to RESULT.txt
Based on the comment the command should be:
pv "RESULT-1.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
You can use for example for loop like this:
for i in Segment??
do
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
done
This will exec the command with all Segment<symbol><symbol> files and ADD result to RESULT.txt
Based on the comment the command should be:
pv "RESULT-1.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
edited Feb 8 at 16:59
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
answered Feb 8 at 6:56
Romeo NinovRomeo Ninov
1,87321014
1,87321014
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
1
Right. I use such way to restrict the filenames. You can use alsoSegment*if you are sure only files in question start with this string
– Romeo Ninov
Feb 8 at 7:13
1
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
1
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
|
show 8 more comments
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
1
Right. I use such way to restrict the filenames. You can use alsoSegment*if you are sure only files in question start with this string
– Romeo Ninov
Feb 8 at 7:13
1
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
1
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
– Joney Walker
Feb 8 at 7:11
1
1
Right. I use such way to restrict the filenames. You can use also
Segment* if you are sure only files in question start with this string– Romeo Ninov
Feb 8 at 7:13
Right. I use such way to restrict the filenames. You can use also
Segment* if you are sure only files in question start with this string– Romeo Ninov
Feb 8 at 7:13
1
1
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
Got it. Thanks a lot.
– Joney Walker
Feb 8 at 7:30
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
– Joney Walker
Feb 8 at 16:56
1
1
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
Ok Sir, I will create a New Question and Tag you in that...
– Joney Walker
Feb 8 at 17:22
|
show 8 more comments
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1403418%2fhow-to-create-a-loop-in-shell-script-to-do-a-specific-task%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
