YARN 3 and Spark: allocate a GPU to executors only
I can't find the Spark option to require executors with a GPU.
I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark.
So far I'm trying out my setup with a minimal cluster (1 resource manager and 2 node managers (each with 8cores, 32Gb RAM, 1 Nvidia GPU), everybody running Ubuntu 18.04.
Resource discovery is working as expected (I see my 16 cores, 56Gb memory and 2 yarn.io/gpu)
I'm able to allocate GPUs while testing yarn jobs with yarn.io/gpu=1, but I'm struggling to do the same with spark-submit.
The documentation provides a way, by using "--conf spark.yarn.executor.resource.yarn.io/gpu=1" but this does not work for me (no effect at all, both in spark-submit command parameter or in $SPARK_CONF/metrics.properties).
The only workaround that I found so far was to set "yarn.resource-types.yarn.io/gpu.minimum-allocation" to 1 in resource-types.xml under $HADOOP_CONF_DIR, but this allocate a GPU to the Master as well, when I only want to target executors.
As YARN 3 is the first one to provide GPU isolation, I try to avoid a rollback to an older(/more documented) version.
I guess this could be set in code through SparkContext and would be happy to know how, but as I'm more on the admin side than ML engineer, I rather set this in conf files once and for all. Anyway at this point, any solution would be appreciated.
Anyone happy to provide the good syntax to allocate GPU with resources isolation enabled ?
Love you guys,
Kevin
gpu cluster hadoop
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
add a comment |
I can't find the Spark option to require executors with a GPU.
I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark.
So far I'm trying out my setup with a minimal cluster (1 resource manager and 2 node managers (each with 8cores, 32Gb RAM, 1 Nvidia GPU), everybody running Ubuntu 18.04.
Resource discovery is working as expected (I see my 16 cores, 56Gb memory and 2 yarn.io/gpu)
I'm able to allocate GPUs while testing yarn jobs with yarn.io/gpu=1, but I'm struggling to do the same with spark-submit.
The documentation provides a way, by using "--conf spark.yarn.executor.resource.yarn.io/gpu=1" but this does not work for me (no effect at all, both in spark-submit command parameter or in $SPARK_CONF/metrics.properties).
The only workaround that I found so far was to set "yarn.resource-types.yarn.io/gpu.minimum-allocation" to 1 in resource-types.xml under $HADOOP_CONF_DIR, but this allocate a GPU to the Master as well, when I only want to target executors.
As YARN 3 is the first one to provide GPU isolation, I try to avoid a rollback to an older(/more documented) version.
I guess this could be set in code through SparkContext and would be happy to know how, but as I'm more on the admin side than ML engineer, I rather set this in conf files once and for all. Anyway at this point, any solution would be appreciated.
Anyone happy to provide the good syntax to allocate GPU with resources isolation enabled ?
Love you guys,
Kevin
gpu cluster hadoop
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
add a comment |
I can't find the Spark option to require executors with a GPU.
I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark.
So far I'm trying out my setup with a minimal cluster (1 resource manager and 2 node managers (each with 8cores, 32Gb RAM, 1 Nvidia GPU), everybody running Ubuntu 18.04.
Resource discovery is working as expected (I see my 16 cores, 56Gb memory and 2 yarn.io/gpu)
I'm able to allocate GPUs while testing yarn jobs with yarn.io/gpu=1, but I'm struggling to do the same with spark-submit.
The documentation provides a way, by using "--conf spark.yarn.executor.resource.yarn.io/gpu=1" but this does not work for me (no effect at all, both in spark-submit command parameter or in $SPARK_CONF/metrics.properties).
The only workaround that I found so far was to set "yarn.resource-types.yarn.io/gpu.minimum-allocation" to 1 in resource-types.xml under $HADOOP_CONF_DIR, but this allocate a GPU to the Master as well, when I only want to target executors.
As YARN 3 is the first one to provide GPU isolation, I try to avoid a rollback to an older(/more documented) version.
I guess this could be set in code through SparkContext and would be happy to know how, but as I'm more on the admin side than ML engineer, I rather set this in conf files once and for all. Anyway at this point, any solution would be appreciated.
Anyone happy to provide the good syntax to allocate GPU with resources isolation enabled ?
Love you guys,
Kevin
gpu cluster hadoop
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
I can't find the Spark option to require executors with a GPU.
I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark.
So far I'm trying out my setup with a minimal cluster (1 resource manager and 2 node managers (each with 8cores, 32Gb RAM, 1 Nvidia GPU), everybody running Ubuntu 18.04.
Resource discovery is working as expected (I see my 16 cores, 56Gb memory and 2 yarn.io/gpu)
I'm able to allocate GPUs while testing yarn jobs with yarn.io/gpu=1, but I'm struggling to do the same with spark-submit.
The documentation provides a way, by using "--conf spark.yarn.executor.resource.yarn.io/gpu=1" but this does not work for me (no effect at all, both in spark-submit command parameter or in $SPARK_CONF/metrics.properties).
The only workaround that I found so far was to set "yarn.resource-types.yarn.io/gpu.minimum-allocation" to 1 in resource-types.xml under $HADOOP_CONF_DIR, but this allocate a GPU to the Master as well, when I only want to target executors.
As YARN 3 is the first one to provide GPU isolation, I try to avoid a rollback to an older(/more documented) version.
I guess this could be set in code through SparkContext and would be happy to know how, but as I'm more on the admin side than ML engineer, I rather set this in conf files once and for all. Anyway at this point, any solution would be appreciated.
Anyone happy to provide the good syntax to allocate GPU with resources isolation enabled ?
Love you guys,
Kevin
gpu cluster hadoop
gpu cluster hadoop
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
edited Dec 28 '18 at 14:23
Kévin Azoulay
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
asked Dec 24 '18 at 16:35
Kévin AzoulayKévin Azoulay
63
63
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Working trick for now:
Set GPU isolation settings in yarn-site and consorts and run spark-submit in cluster mode without GPU.
The Spark driver will run inside the AM container instantiated.
From within your Spark app, through SparkContext, now set yarn.io/gpu.minimum-allocation to 1.
This will take effect for the next containers allocations: your workers !
Ugly, but efficient trusting my first tests.
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1387424%2fyarn-3-and-spark-allocate-a-gpu-to-executors-only%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Working trick for now:
Set GPU isolation settings in yarn-site and consorts and run spark-submit in cluster mode without GPU.
The Spark driver will run inside the AM container instantiated.
From within your Spark app, through SparkContext, now set yarn.io/gpu.minimum-allocation to 1.
This will take effect for the next containers allocations: your workers !
Ugly, but efficient trusting my first tests.
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
add a comment |
Working trick for now:
Set GPU isolation settings in yarn-site and consorts and run spark-submit in cluster mode without GPU.
The Spark driver will run inside the AM container instantiated.
From within your Spark app, through SparkContext, now set yarn.io/gpu.minimum-allocation to 1.
This will take effect for the next containers allocations: your workers !
Ugly, but efficient trusting my first tests.
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
add a comment |
Working trick for now:
Set GPU isolation settings in yarn-site and consorts and run spark-submit in cluster mode without GPU.
The Spark driver will run inside the AM container instantiated.
From within your Spark app, through SparkContext, now set yarn.io/gpu.minimum-allocation to 1.
This will take effect for the next containers allocations: your workers !
Ugly, but efficient trusting my first tests.
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
Working trick for now:
Set GPU isolation settings in yarn-site and consorts and run spark-submit in cluster mode without GPU.
The Spark driver will run inside the AM container instantiated.
From within your Spark app, through SparkContext, now set yarn.io/gpu.minimum-allocation to 1.
This will take effect for the next containers allocations: your workers !
Ugly, but efficient trusting my first tests.
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
answered Jan 4 at 10:27
Kévin AzoulayKévin Azoulay
63
63
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1387424%2fyarn-3-and-spark-allocate-a-gpu-to-executors-only%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
