Data corruption on copying: large files and small files
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I know that files sometimes corrupted during copying, and for that reason people check them with hash sums. (MD5 or CRC32 seems to be OK for random errors).
But usually, when I read about this use case, people talk about large files. My assumption is that such corruption doesn't usually affect small files, only big ones. Is this assumption is correct?
data-transfer hashing
asked Mar 11 at 9:42
john c. j.john c. j.
17414
add a comment |
I know that files sometimes corrupted during copying, and for that reason people check them with hash sums. (MD5 or CRC32 seems to be OK for random errors).
But usually, when I read about this use case, people talk about large files. My assumption is that such corruption doesn't usually affect small files, only big ones. Is this assumption is correct?
data-transfer hashing
asked Mar 11 at 9:42
john c. j.john c. j.
17414
add a comment |
I know that files sometimes corrupted during copying, and for that reason people check them with hash sums. (MD5 or CRC32 seems to be OK for random errors).
But usually, when I read about this use case, people talk about large files. My assumption is that such corruption doesn't usually affect small files, only big ones. Is this assumption is correct?
data-transfer hashing
asked Mar 11 at 9:42
john c. j.john c. j.
17414
I know that files sometimes corrupted during copying, and for that reason people check them with hash sums. (MD5 or CRC32 seems to be OK for random errors).
But usually, when I read about this use case, people talk about large files. My assumption is that such corruption doesn't usually affect small files, only big ones. Is this assumption is correct?
data-transfer hashing
data-transfer hashing
asked Mar 11 at 9:42
john c. j.john c. j.
17414
asked Mar 11 at 9:42
john c. j.john c. j.
17414
asked Mar 11 at 9:42
john c. j.john c. j.
17414
asked Mar 11 at 9:42
john c. j.john c. j.
17414
asked Mar 11 at 9:42
john c. j.john c. j.
17414
17414
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Larger files have more opportunities for the corruption to occur.
Let's say that probability of a single byte becoming corrupted is 0.01% (top of the hat number, just for the purpose of this demonstration).
- Probability of a 1 B file being error-free is 99.99% (100% - 0.01%)
- Probability of a 100 B file being error-free is approx. 99.00% (99.99%100)
- Probability of a 1 KiB (1024 B) file being error-free is approx. 90.27%
- Probability of a 10 KiB file being error-free is approx. 35.91%
- Probability of a 1 MiB (1024 KiB) file being error-free is approx. 0%
As you can see, probabilities of errors accumulate very rapidly as the size increases. That's why large files are way more likely to be corrupted. This doesn't mean that small files are not affected by corruption, though.
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
1
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1413031%2fdata-corruption-on-copying-large-files-and-small-files%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Larger files have more opportunities for the corruption to occur.
Let's say that probability of a single byte becoming corrupted is 0.01% (top of the hat number, just for the purpose of this demonstration).
- Probability of a 1 B file being error-free is 99.99% (100% - 0.01%)
- Probability of a 100 B file being error-free is approx. 99.00% (99.99%100)
- Probability of a 1 KiB (1024 B) file being error-free is approx. 90.27%
- Probability of a 10 KiB file being error-free is approx. 35.91%
- Probability of a 1 MiB (1024 KiB) file being error-free is approx. 0%
As you can see, probabilities of errors accumulate very rapidly as the size increases. That's why large files are way more likely to be corrupted. This doesn't mean that small files are not affected by corruption, though.
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
1
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
add a comment |
Larger files have more opportunities for the corruption to occur.
Let's say that probability of a single byte becoming corrupted is 0.01% (top of the hat number, just for the purpose of this demonstration).
- Probability of a 1 B file being error-free is 99.99% (100% - 0.01%)
- Probability of a 100 B file being error-free is approx. 99.00% (99.99%100)
- Probability of a 1 KiB (1024 B) file being error-free is approx. 90.27%
- Probability of a 10 KiB file being error-free is approx. 35.91%
- Probability of a 1 MiB (1024 KiB) file being error-free is approx. 0%
As you can see, probabilities of errors accumulate very rapidly as the size increases. That's why large files are way more likely to be corrupted. This doesn't mean that small files are not affected by corruption, though.
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
1
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
add a comment |
Larger files have more opportunities for the corruption to occur.
Let's say that probability of a single byte becoming corrupted is 0.01% (top of the hat number, just for the purpose of this demonstration).
- Probability of a 1 B file being error-free is 99.99% (100% - 0.01%)
- Probability of a 100 B file being error-free is approx. 99.00% (99.99%100)
- Probability of a 1 KiB (1024 B) file being error-free is approx. 90.27%
- Probability of a 10 KiB file being error-free is approx. 35.91%
- Probability of a 1 MiB (1024 KiB) file being error-free is approx. 0%
As you can see, probabilities of errors accumulate very rapidly as the size increases. That's why large files are way more likely to be corrupted. This doesn't mean that small files are not affected by corruption, though.
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
Larger files have more opportunities for the corruption to occur.
Let's say that probability of a single byte becoming corrupted is 0.01% (top of the hat number, just for the purpose of this demonstration).
- Probability of a 1 B file being error-free is 99.99% (100% - 0.01%)
- Probability of a 100 B file being error-free is approx. 99.00% (99.99%100)
- Probability of a 1 KiB (1024 B) file being error-free is approx. 90.27%
- Probability of a 10 KiB file being error-free is approx. 35.91%
- Probability of a 1 MiB (1024 KiB) file being error-free is approx. 0%
As you can see, probabilities of errors accumulate very rapidly as the size increases. That's why large files are way more likely to be corrupted. This doesn't mean that small files are not affected by corruption, though.
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
answered Mar 11 at 10:04
gronostajgronostaj
29.3k1472108
29.3k1472108
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
1
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
add a comment |
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
1
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
Thanks. Probably you know: does hash sums allow to tell if file has become corrupted due data degradation?
– john c. j.
Mar 11 at 14:32
1
1
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
Hashes can be used to detect if perceivable content of file has changed: for the same content, the hash is always the same, but even little changes will cause good hashes to change drastically. The cause of modification is irrelevant and not detectable by hash itself. It can be data decay, hardware malfunction, transfer error, virus or intentional modification by user, doesn't matter. If the content has changed, so did the hash.
– gronostaj
Mar 11 at 20:33
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1413031%2fdata-corruption-on-copying-large-files-and-small-files%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
