Using scp to get all files in multiple directories
I have a directory structure which looks like A1/B1/C1/files111*
, A1/B1/C2/files112*
, A1/B2/C1/files121*
, A1/B2/C2/files122*
, etc. so that all files in all directories are unique. Is there a way to use scp (or some other tool) to pull all terminal files files###*
at once?
linux scp
add a comment |
I have a directory structure which looks like A1/B1/C1/files111*
, A1/B1/C2/files112*
, A1/B2/C1/files121*
, A1/B2/C2/files122*
, etc. so that all files in all directories are unique. Is there a way to use scp (or some other tool) to pull all terminal files files###*
at once?
linux scp
add a comment |
I have a directory structure which looks like A1/B1/C1/files111*
, A1/B1/C2/files112*
, A1/B2/C1/files121*
, A1/B2/C2/files122*
, etc. so that all files in all directories are unique. Is there a way to use scp (or some other tool) to pull all terminal files files###*
at once?
linux scp
I have a directory structure which looks like A1/B1/C1/files111*
, A1/B1/C2/files112*
, A1/B2/C1/files121*
, A1/B2/C2/files122*
, etc. so that all files in all directories are unique. Is there a way to use scp (or some other tool) to pull all terminal files files###*
at once?
linux scp
linux scp
asked May 4 '15 at 20:09
drjrm3drjrm3
62921732
62921732
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
Since you didn't specify this in the question, I will presume that the directory structure is remote, and you want it local.
What you need to do is use wildcards, but escape them on the client side, so that your local shell ignores them and passes them on. Combine that with a recursive flag (-r
), and it should perform how you want it:
scp -r remotemachine:some/directory/files* /some/local/target/directory/
The important part is the backslash, as that makes your local shell ignore the asterisk, and pass it on instead. This should allow you to pull all files*
from remote, including subdirectories. It will also pull any actual files whos name begin with files*
.
add a comment |
First you have to consider what you want to happen to the files that are copied in this way, and how it may affect future copy operations. Are they going to stay there? Will they be deleted? Archived? Once you know this you can build your copy command, and then the necessary cleanup operations. Here's a sample script that first copies all existing files, then deletes them from the remote server.
There's a risk the files may change between copy and delete operations, so I'd first move them to a temp dir, then copy and delete them from there, allowing the working directory to continue regular operation.
I'd also create a cron job to retrieve these files automatically at the same time each day, and add some sanity checks to ensure the operations are completed successfully.
#!/bin/bash
HOST="RemoteServerIPorHostName"
localDir="/some/local/dir"
for remoteTargets in "/A1/B1/C1/files111*" "/A1/B1/C2/files112*" "/A1/B2/C1/files121*" "/A1/B2/C2/files122*"
do
scp -pr user@$HOST:$remoteTargets $localDir
ssh -t user@$HOST 'rm -rf $remoteTargets'
done
add a comment |
ssh user@remote "find /target/A1 -iname "*files*" -exec scp {} user@local:/desitnation ;"
ssh
connects to a remote machine and execute code enclosed by the double quotes""
find
searchs inside of/target/A1
directory for all files containing the keyword "files". Please note you need the backslashes for case insensitive search parameter-iname
and the double quotes in order to avoid leaving the previous pair too early.
-exec
is anotherfind
parameter, used to execute code. In our case we executescp
and pass all search results fromfind
toscp
with the curved brackets{}
.
scp
copies your found files to your local or any other machine.
This worked in my test setup as I have exchanged the public keys in order to use authentication without a password.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f909730%2fusing-scp-to-get-all-files-in-multiple-directories%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
Since you didn't specify this in the question, I will presume that the directory structure is remote, and you want it local.
What you need to do is use wildcards, but escape them on the client side, so that your local shell ignores them and passes them on. Combine that with a recursive flag (-r
), and it should perform how you want it:
scp -r remotemachine:some/directory/files* /some/local/target/directory/
The important part is the backslash, as that makes your local shell ignore the asterisk, and pass it on instead. This should allow you to pull all files*
from remote, including subdirectories. It will also pull any actual files whos name begin with files*
.
add a comment |
Since you didn't specify this in the question, I will presume that the directory structure is remote, and you want it local.
What you need to do is use wildcards, but escape them on the client side, so that your local shell ignores them and passes them on. Combine that with a recursive flag (-r
), and it should perform how you want it:
scp -r remotemachine:some/directory/files* /some/local/target/directory/
The important part is the backslash, as that makes your local shell ignore the asterisk, and pass it on instead. This should allow you to pull all files*
from remote, including subdirectories. It will also pull any actual files whos name begin with files*
.
add a comment |
Since you didn't specify this in the question, I will presume that the directory structure is remote, and you want it local.
What you need to do is use wildcards, but escape them on the client side, so that your local shell ignores them and passes them on. Combine that with a recursive flag (-r
), and it should perform how you want it:
scp -r remotemachine:some/directory/files* /some/local/target/directory/
The important part is the backslash, as that makes your local shell ignore the asterisk, and pass it on instead. This should allow you to pull all files*
from remote, including subdirectories. It will also pull any actual files whos name begin with files*
.
Since you didn't specify this in the question, I will presume that the directory structure is remote, and you want it local.
What you need to do is use wildcards, but escape them on the client side, so that your local shell ignores them and passes them on. Combine that with a recursive flag (-r
), and it should perform how you want it:
scp -r remotemachine:some/directory/files* /some/local/target/directory/
The important part is the backslash, as that makes your local shell ignore the asterisk, and pass it on instead. This should allow you to pull all files*
from remote, including subdirectories. It will also pull any actual files whos name begin with files*
.
answered May 4 '15 at 20:20
JarmundJarmund
4,62952148
4,62952148
add a comment |
add a comment |
First you have to consider what you want to happen to the files that are copied in this way, and how it may affect future copy operations. Are they going to stay there? Will they be deleted? Archived? Once you know this you can build your copy command, and then the necessary cleanup operations. Here's a sample script that first copies all existing files, then deletes them from the remote server.
There's a risk the files may change between copy and delete operations, so I'd first move them to a temp dir, then copy and delete them from there, allowing the working directory to continue regular operation.
I'd also create a cron job to retrieve these files automatically at the same time each day, and add some sanity checks to ensure the operations are completed successfully.
#!/bin/bash
HOST="RemoteServerIPorHostName"
localDir="/some/local/dir"
for remoteTargets in "/A1/B1/C1/files111*" "/A1/B1/C2/files112*" "/A1/B2/C1/files121*" "/A1/B2/C2/files122*"
do
scp -pr user@$HOST:$remoteTargets $localDir
ssh -t user@$HOST 'rm -rf $remoteTargets'
done
add a comment |
First you have to consider what you want to happen to the files that are copied in this way, and how it may affect future copy operations. Are they going to stay there? Will they be deleted? Archived? Once you know this you can build your copy command, and then the necessary cleanup operations. Here's a sample script that first copies all existing files, then deletes them from the remote server.
There's a risk the files may change between copy and delete operations, so I'd first move them to a temp dir, then copy and delete them from there, allowing the working directory to continue regular operation.
I'd also create a cron job to retrieve these files automatically at the same time each day, and add some sanity checks to ensure the operations are completed successfully.
#!/bin/bash
HOST="RemoteServerIPorHostName"
localDir="/some/local/dir"
for remoteTargets in "/A1/B1/C1/files111*" "/A1/B1/C2/files112*" "/A1/B2/C1/files121*" "/A1/B2/C2/files122*"
do
scp -pr user@$HOST:$remoteTargets $localDir
ssh -t user@$HOST 'rm -rf $remoteTargets'
done
add a comment |
First you have to consider what you want to happen to the files that are copied in this way, and how it may affect future copy operations. Are they going to stay there? Will they be deleted? Archived? Once you know this you can build your copy command, and then the necessary cleanup operations. Here's a sample script that first copies all existing files, then deletes them from the remote server.
There's a risk the files may change between copy and delete operations, so I'd first move them to a temp dir, then copy and delete them from there, allowing the working directory to continue regular operation.
I'd also create a cron job to retrieve these files automatically at the same time each day, and add some sanity checks to ensure the operations are completed successfully.
#!/bin/bash
HOST="RemoteServerIPorHostName"
localDir="/some/local/dir"
for remoteTargets in "/A1/B1/C1/files111*" "/A1/B1/C2/files112*" "/A1/B2/C1/files121*" "/A1/B2/C2/files122*"
do
scp -pr user@$HOST:$remoteTargets $localDir
ssh -t user@$HOST 'rm -rf $remoteTargets'
done
First you have to consider what you want to happen to the files that are copied in this way, and how it may affect future copy operations. Are they going to stay there? Will they be deleted? Archived? Once you know this you can build your copy command, and then the necessary cleanup operations. Here's a sample script that first copies all existing files, then deletes them from the remote server.
There's a risk the files may change between copy and delete operations, so I'd first move them to a temp dir, then copy and delete them from there, allowing the working directory to continue regular operation.
I'd also create a cron job to retrieve these files automatically at the same time each day, and add some sanity checks to ensure the operations are completed successfully.
#!/bin/bash
HOST="RemoteServerIPorHostName"
localDir="/some/local/dir"
for remoteTargets in "/A1/B1/C1/files111*" "/A1/B1/C2/files112*" "/A1/B2/C1/files121*" "/A1/B2/C2/files122*"
do
scp -pr user@$HOST:$remoteTargets $localDir
ssh -t user@$HOST 'rm -rf $remoteTargets'
done
answered May 4 '15 at 20:58
Alex AtkinsonAlex Atkinson
2,735913
2,735913
add a comment |
add a comment |
ssh user@remote "find /target/A1 -iname "*files*" -exec scp {} user@local:/desitnation ;"
ssh
connects to a remote machine and execute code enclosed by the double quotes""
find
searchs inside of/target/A1
directory for all files containing the keyword "files". Please note you need the backslashes for case insensitive search parameter-iname
and the double quotes in order to avoid leaving the previous pair too early.
-exec
is anotherfind
parameter, used to execute code. In our case we executescp
and pass all search results fromfind
toscp
with the curved brackets{}
.
scp
copies your found files to your local or any other machine.
This worked in my test setup as I have exchanged the public keys in order to use authentication without a password.
add a comment |
ssh user@remote "find /target/A1 -iname "*files*" -exec scp {} user@local:/desitnation ;"
ssh
connects to a remote machine and execute code enclosed by the double quotes""
find
searchs inside of/target/A1
directory for all files containing the keyword "files". Please note you need the backslashes for case insensitive search parameter-iname
and the double quotes in order to avoid leaving the previous pair too early.
-exec
is anotherfind
parameter, used to execute code. In our case we executescp
and pass all search results fromfind
toscp
with the curved brackets{}
.
scp
copies your found files to your local or any other machine.
This worked in my test setup as I have exchanged the public keys in order to use authentication without a password.
add a comment |
ssh user@remote "find /target/A1 -iname "*files*" -exec scp {} user@local:/desitnation ;"
ssh
connects to a remote machine and execute code enclosed by the double quotes""
find
searchs inside of/target/A1
directory for all files containing the keyword "files". Please note you need the backslashes for case insensitive search parameter-iname
and the double quotes in order to avoid leaving the previous pair too early.
-exec
is anotherfind
parameter, used to execute code. In our case we executescp
and pass all search results fromfind
toscp
with the curved brackets{}
.
scp
copies your found files to your local or any other machine.
This worked in my test setup as I have exchanged the public keys in order to use authentication without a password.
ssh user@remote "find /target/A1 -iname "*files*" -exec scp {} user@local:/desitnation ;"
ssh
connects to a remote machine and execute code enclosed by the double quotes""
find
searchs inside of/target/A1
directory for all files containing the keyword "files". Please note you need the backslashes for case insensitive search parameter-iname
and the double quotes in order to avoid leaving the previous pair too early.
-exec
is anotherfind
parameter, used to execute code. In our case we executescp
and pass all search results fromfind
toscp
with the curved brackets{}
.
scp
copies your found files to your local or any other machine.
This worked in my test setup as I have exchanged the public keys in order to use authentication without a password.
edited May 6 '15 at 8:38
answered May 4 '15 at 20:46
XylolXylol
112
112
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f909730%2fusing-scp-to-get-all-files-in-multiple-directories%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown