How do I optimize Windows disk I/O for high-latency throughput?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
Let's say there are 25 processes each which want to read and index a large (say, 1GB) file.
The hard drive may be capable of ~100MB/sec throughput, but the head movement and rotational latency involved in moving between such processes is significant so may reduce throughput to 5MB/sec, or just 5% of the conventional drive's throughput capability.
Were the hard drive to read a large amount of data (say, 32MB) or read whatever it can for a long time (say, 200ms) before moving onto the next process, file IO could approach the full throughput of the drive at the expense of latency.
Does Windows have such a facility for accomplishing this goal in cases where latency takes a back seat to throughput?
hard-drive performance latency windows-server-2012-r2 throughput
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
add a comment |
Let's say there are 25 processes each which want to read and index a large (say, 1GB) file.
The hard drive may be capable of ~100MB/sec throughput, but the head movement and rotational latency involved in moving between such processes is significant so may reduce throughput to 5MB/sec, or just 5% of the conventional drive's throughput capability.
Were the hard drive to read a large amount of data (say, 32MB) or read whatever it can for a long time (say, 200ms) before moving onto the next process, file IO could approach the full throughput of the drive at the expense of latency.
Does Windows have such a facility for accomplishing this goal in cases where latency takes a back seat to throughput?
hard-drive performance latency windows-server-2012-r2 throughput
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
check the tool DiskSpd: blogs.technet.com/b/josebda/archive/2014/10/13/…
– magicandre1981
Nov 1 '14 at 7:11
add a comment |
Let's say there are 25 processes each which want to read and index a large (say, 1GB) file.
The hard drive may be capable of ~100MB/sec throughput, but the head movement and rotational latency involved in moving between such processes is significant so may reduce throughput to 5MB/sec, or just 5% of the conventional drive's throughput capability.
Were the hard drive to read a large amount of data (say, 32MB) or read whatever it can for a long time (say, 200ms) before moving onto the next process, file IO could approach the full throughput of the drive at the expense of latency.
Does Windows have such a facility for accomplishing this goal in cases where latency takes a back seat to throughput?
hard-drive performance latency windows-server-2012-r2 throughput
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
Let's say there are 25 processes each which want to read and index a large (say, 1GB) file.
The hard drive may be capable of ~100MB/sec throughput, but the head movement and rotational latency involved in moving between such processes is significant so may reduce throughput to 5MB/sec, or just 5% of the conventional drive's throughput capability.
Were the hard drive to read a large amount of data (say, 32MB) or read whatever it can for a long time (say, 200ms) before moving onto the next process, file IO could approach the full throughput of the drive at the expense of latency.
Does Windows have such a facility for accomplishing this goal in cases where latency takes a back seat to throughput?
hard-drive performance latency windows-server-2012-r2 throughput
hard-drive performance latency windows-server-2012-r2 throughput
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
asked Nov 1 '14 at 4:38
Charles BurnsCharles Burns
5701611
5701611
check the tool DiskSpd: blogs.technet.com/b/josebda/archive/2014/10/13/…
– magicandre1981
Nov 1 '14 at 7:11
add a comment |
check the tool DiskSpd: blogs.technet.com/b/josebda/archive/2014/10/13/…
– magicandre1981
Nov 1 '14 at 7:11
check the tool DiskSpd: blogs.technet.com/b/josebda/archive/2014/10/13/…
– magicandre1981
Nov 1 '14 at 7:11
check the tool DiskSpd: blogs.technet.com/b/josebda/archive/2014/10/13/…
– magicandre1981
Nov 1 '14 at 7:11
add a comment |
1 Answer
1
active
oldest
votes
This is what the NCQ protocol is for. A drive that supports NCQ looks at all of the incoming commands and reorders them to service them more efficiently (and essentially, reordering the commands is what you're asking about).
That said, I see in your example you have a disk queue depth of 27, so we're well past the help of even NCQ. This is a problem for RAID (possibly coupled with faster physical storage).
There are storage parameters that can be adjusted in Windows, particularly those related to caching behavior, but that's not going to help enough in your case.
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f834537%2fhow-do-i-optimize-windows-disk-i-o-for-high-latency-throughput%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This is what the NCQ protocol is for. A drive that supports NCQ looks at all of the incoming commands and reorders them to service them more efficiently (and essentially, reordering the commands is what you're asking about).
That said, I see in your example you have a disk queue depth of 27, so we're well past the help of even NCQ. This is a problem for RAID (possibly coupled with faster physical storage).
There are storage parameters that can be adjusted in Windows, particularly those related to caching behavior, but that's not going to help enough in your case.
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
add a comment |
This is what the NCQ protocol is for. A drive that supports NCQ looks at all of the incoming commands and reorders them to service them more efficiently (and essentially, reordering the commands is what you're asking about).
That said, I see in your example you have a disk queue depth of 27, so we're well past the help of even NCQ. This is a problem for RAID (possibly coupled with faster physical storage).
There are storage parameters that can be adjusted in Windows, particularly those related to caching behavior, but that's not going to help enough in your case.
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
add a comment |
This is what the NCQ protocol is for. A drive that supports NCQ looks at all of the incoming commands and reorders them to service them more efficiently (and essentially, reordering the commands is what you're asking about).
That said, I see in your example you have a disk queue depth of 27, so we're well past the help of even NCQ. This is a problem for RAID (possibly coupled with faster physical storage).
There are storage parameters that can be adjusted in Windows, particularly those related to caching behavior, but that's not going to help enough in your case.
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
This is what the NCQ protocol is for. A drive that supports NCQ looks at all of the incoming commands and reorders them to service them more efficiently (and essentially, reordering the commands is what you're asking about).
That said, I see in your example you have a disk queue depth of 27, so we're well past the help of even NCQ. This is a problem for RAID (possibly coupled with faster physical storage).
There are storage parameters that can be adjusted in Windows, particularly those related to caching behavior, but that's not going to help enough in your case.
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
edited Nov 2 '14 at 2:59
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
answered Nov 1 '14 at 12:28
Twisty ImpersonatorTwisty Impersonator
18.7k1468100
18.7k1468100
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
add a comment |
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
I think of NCQ as more for helping small, near random I/Os like database access. The drive involved does have NCQ and it probably helps a bit, bit I think this situation is more about tuning the Windows I/O scheduler. Linux allows me to choose I/O schedulers (deadline, CFQ...), but in fairness none of them would be much better than Windows' default in this case.
– Charles Burns
Nov 1 '14 at 14:28
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
Additionally NCQ is in use by the drive whose performance is tested in the original question, still yielding less than 5% of the maximum throughput.
– Charles Burns
Apr 14 '16 at 19:02
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f834537%2fhow-do-i-optimize-windows-disk-i-o-for-high-latency-throughput%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
check the tool DiskSpd: blogs.technet.com/b/josebda/archive/2014/10/13/…
– magicandre1981
Nov 1 '14 at 7:11