Very slow server performance at low loads
I use a server at work to run models on modestly large datasets in RAM (think 10GB to 100GB). There are only a few people on this server at any time. The server has lots of RAM (over 1TB) and many processors. We have found that when RAM usage exceeds what seems to be a modest threshold - think three people loading a combined 100GB of data into RAM in applications like R or Stata - then the server becomes dramatically slower. Operations that would take seconds on my PC at home take hours or days on the server. I am not sure why this would be the case: it is as if the server doesn't want to free cached memory, and even operating on the data currently held in memory takes an insanely long amount of time (CPU load is low: <10%). Even stuff on the command line takes a while: listing files can take a few seconds, etc. I don't have permissions to edit things on the server myself. Does anyone know what could be going on here, or things I can hunt for without having root access? The system administrators don't know what is going on.
We are running Red Hat Enterprise 6.9.
Thanks so much for your help!
linux memory performance memory-management
asked Feb 16 at 2:10
framseyframsey
1
add a comment |
I use a server at work to run models on modestly large datasets in RAM (think 10GB to 100GB). There are only a few people on this server at any time. The server has lots of RAM (over 1TB) and many processors. We have found that when RAM usage exceeds what seems to be a modest threshold - think three people loading a combined 100GB of data into RAM in applications like R or Stata - then the server becomes dramatically slower. Operations that would take seconds on my PC at home take hours or days on the server. I am not sure why this would be the case: it is as if the server doesn't want to free cached memory, and even operating on the data currently held in memory takes an insanely long amount of time (CPU load is low: <10%). Even stuff on the command line takes a while: listing files can take a few seconds, etc. I don't have permissions to edit things on the server myself. Does anyone know what could be going on here, or things I can hunt for without having root access? The system administrators don't know what is going on.
We are running Red Hat Enterprise 6.9.
Thanks so much for your help!
linux memory performance memory-management
asked Feb 16 at 2:10
framseyframsey
1
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing).
– Spiff
Feb 16 at 2:27
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously).
– framsey
Feb 16 at 5:03
add a comment |
I use a server at work to run models on modestly large datasets in RAM (think 10GB to 100GB). There are only a few people on this server at any time. The server has lots of RAM (over 1TB) and many processors. We have found that when RAM usage exceeds what seems to be a modest threshold - think three people loading a combined 100GB of data into RAM in applications like R or Stata - then the server becomes dramatically slower. Operations that would take seconds on my PC at home take hours or days on the server. I am not sure why this would be the case: it is as if the server doesn't want to free cached memory, and even operating on the data currently held in memory takes an insanely long amount of time (CPU load is low: <10%). Even stuff on the command line takes a while: listing files can take a few seconds, etc. I don't have permissions to edit things on the server myself. Does anyone know what could be going on here, or things I can hunt for without having root access? The system administrators don't know what is going on.
We are running Red Hat Enterprise 6.9.
Thanks so much for your help!
linux memory performance memory-management
asked Feb 16 at 2:10
framseyframsey
1
I use a server at work to run models on modestly large datasets in RAM (think 10GB to 100GB). There are only a few people on this server at any time. The server has lots of RAM (over 1TB) and many processors. We have found that when RAM usage exceeds what seems to be a modest threshold - think three people loading a combined 100GB of data into RAM in applications like R or Stata - then the server becomes dramatically slower. Operations that would take seconds on my PC at home take hours or days on the server. I am not sure why this would be the case: it is as if the server doesn't want to free cached memory, and even operating on the data currently held in memory takes an insanely long amount of time (CPU load is low: <10%). Even stuff on the command line takes a while: listing files can take a few seconds, etc. I don't have permissions to edit things on the server myself. Does anyone know what could be going on here, or things I can hunt for without having root access? The system administrators don't know what is going on.
We are running Red Hat Enterprise 6.9.
Thanks so much for your help!
linux memory performance memory-management
linux memory performance memory-management
asked Feb 16 at 2:10
framseyframsey
1
asked Feb 16 at 2:10
framseyframsey
1
asked Feb 16 at 2:10
framseyframsey
1
asked Feb 16 at 2:10
framseyframsey
1
asked Feb 16 at 2:10
framseyframsey
1
1
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing).
– Spiff
Feb 16 at 2:27
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously).
– framsey
Feb 16 at 5:03
add a comment |
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing).
– Spiff
Feb 16 at 2:27
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously).
– framsey
Feb 16 at 5:03
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing).
– Spiff
Feb 16 at 2:27
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing).
– Spiff
Feb 16 at 2:27
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously).
– framsey
Feb 16 at 5:03
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously).
– framsey
Feb 16 at 5:03
add a comment |
1 Answer
1
active
oldest
votes
Here's a few ideas of things to check, some probably require sudo but some hopefully don't (like just cat).
Check if there's any swap with
cat /proc/swaps
swapon -sswapon --show
Check "swappiness" with
cat /proc/sys/vm/swappiness
Try monitoring temperatures or CPU frequencies, maybe something's overheating and/or throttling down.
If ram were overheating, I'd expect a bunch of random errors, unless it's only 1 out of 50 sticks, and I think RAM rarely has temp monitors...
Anything in
dmesgor/var/log/syslog?
Clear/flush the disk caches with
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Maybe there's a ton of disk or network activity, check with a program like
iftoporiotop
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1406347%2fvery-slow-server-performance-at-low-loads%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Here's a few ideas of things to check, some probably require sudo but some hopefully don't (like just cat).
Check if there's any swap with
cat /proc/swaps
swapon -sswapon --show
Check "swappiness" with
cat /proc/sys/vm/swappiness
Try monitoring temperatures or CPU frequencies, maybe something's overheating and/or throttling down.
If ram were overheating, I'd expect a bunch of random errors, unless it's only 1 out of 50 sticks, and I think RAM rarely has temp monitors...
Anything in
dmesgor/var/log/syslog?
Clear/flush the disk caches with
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Maybe there's a ton of disk or network activity, check with a program like
iftoporiotop
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
add a comment |
Here's a few ideas of things to check, some probably require sudo but some hopefully don't (like just cat).
Check if there's any swap with
cat /proc/swaps
swapon -sswapon --show
Check "swappiness" with
cat /proc/sys/vm/swappiness
Try monitoring temperatures or CPU frequencies, maybe something's overheating and/or throttling down.
If ram were overheating, I'd expect a bunch of random errors, unless it's only 1 out of 50 sticks, and I think RAM rarely has temp monitors...
Anything in
dmesgor/var/log/syslog?
Clear/flush the disk caches with
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Maybe there's a ton of disk or network activity, check with a program like
iftoporiotop
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
add a comment |
Here's a few ideas of things to check, some probably require sudo but some hopefully don't (like just cat).
Check if there's any swap with
cat /proc/swaps
swapon -sswapon --show
Check "swappiness" with
cat /proc/sys/vm/swappiness
Try monitoring temperatures or CPU frequencies, maybe something's overheating and/or throttling down.
If ram were overheating, I'd expect a bunch of random errors, unless it's only 1 out of 50 sticks, and I think RAM rarely has temp monitors...
Anything in
dmesgor/var/log/syslog?
Clear/flush the disk caches with
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Maybe there's a ton of disk or network activity, check with a program like
iftoporiotop
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
Here's a few ideas of things to check, some probably require sudo but some hopefully don't (like just cat).
Check if there's any swap with
cat /proc/swaps
swapon -sswapon --show
Check "swappiness" with
cat /proc/sys/vm/swappiness
Try monitoring temperatures or CPU frequencies, maybe something's overheating and/or throttling down.
If ram were overheating, I'd expect a bunch of random errors, unless it's only 1 out of 50 sticks, and I think RAM rarely has temp monitors...
Anything in
dmesgor/var/log/syslog?
Clear/flush the disk caches with
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Maybe there's a ton of disk or network activity, check with a program like
iftoporiotop
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
answered Feb 16 at 5:18
Xen2050Xen2050
11.1k31636
11.1k31636
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1406347%2fvery-slow-server-performance-at-low-loads%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
This feels like it's going to come down to the server not really using all of its RAM and ending up using virtual memory and swapping to disk too much (thrashing).
– Spiff
Feb 16 at 2:27
Do you know how I can check this out, or what settings the system administrators should be tuning? If it helps, there is no swap space as far as I can tell (from top - swap's always at 0). That's probably a bad idea for a few reasons, but we're not getting processes killed because we have way more than enough RAM (except when we hit about 10-15% utilization, things slow down enormously).
– framsey
Feb 16 at 5:03