How does multi cpu system shares pci-express bus?
For a modern single-cpu system things are transparent: there is multicore cpu, where cores are talking via uncore functions (like L3 cache, shared on-die memory controller). Uncore does also all the handling pci-e bus functions, providing access to GPU, network card or ssd for instance. I can read that a particular processor does support 40 pci-e lanes, for example.
But things go weird when I am trying to think of how does two separate processors could talk to a single gpu? How that is usually solved on multi-socket chipsets (Intel® C612 for example)? Can i use 80 pci-e lanes from both processors, to connect 80 gpus, so that every core from any of both chips can talk to every gpu?
pci-express sockets multi-processor numa
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
add a comment |
For a modern single-cpu system things are transparent: there is multicore cpu, where cores are talking via uncore functions (like L3 cache, shared on-die memory controller). Uncore does also all the handling pci-e bus functions, providing access to GPU, network card or ssd for instance. I can read that a particular processor does support 40 pci-e lanes, for example.
But things go weird when I am trying to think of how does two separate processors could talk to a single gpu? How that is usually solved on multi-socket chipsets (Intel® C612 for example)? Can i use 80 pci-e lanes from both processors, to connect 80 gpus, so that every core from any of both chips can talk to every gpu?
pci-express sockets multi-processor numa
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
There is an inter-processor link between the two CPUs for memory access (see NUMA), I would expect access to shared resources such as GPUs to be handled similarly.
– Mokubai♦
Feb 23 at 14:59
@Mokubai So does they work through memory? Or does they have an additional specialized bus for sharing pci-express functions?
– xakepp35
Feb 23 at 15:08
See superuser.com/a/1403322/19943 for an image of how they are connected and QPI
– Mokubai♦
Feb 23 at 15:17
add a comment |
For a modern single-cpu system things are transparent: there is multicore cpu, where cores are talking via uncore functions (like L3 cache, shared on-die memory controller). Uncore does also all the handling pci-e bus functions, providing access to GPU, network card or ssd for instance. I can read that a particular processor does support 40 pci-e lanes, for example.
But things go weird when I am trying to think of how does two separate processors could talk to a single gpu? How that is usually solved on multi-socket chipsets (Intel® C612 for example)? Can i use 80 pci-e lanes from both processors, to connect 80 gpus, so that every core from any of both chips can talk to every gpu?
pci-express sockets multi-processor numa
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
For a modern single-cpu system things are transparent: there is multicore cpu, where cores are talking via uncore functions (like L3 cache, shared on-die memory controller). Uncore does also all the handling pci-e bus functions, providing access to GPU, network card or ssd for instance. I can read that a particular processor does support 40 pci-e lanes, for example.
But things go weird when I am trying to think of how does two separate processors could talk to a single gpu? How that is usually solved on multi-socket chipsets (Intel® C612 for example)? Can i use 80 pci-e lanes from both processors, to connect 80 gpus, so that every core from any of both chips can talk to every gpu?
pci-express sockets multi-processor numa
pci-express sockets multi-processor numa
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
asked Feb 23 at 14:40
xakepp35xakepp35
1701113
1701113
There is an inter-processor link between the two CPUs for memory access (see NUMA), I would expect access to shared resources such as GPUs to be handled similarly.
– Mokubai♦
Feb 23 at 14:59
@Mokubai So does they work through memory? Or does they have an additional specialized bus for sharing pci-express functions?
– xakepp35
Feb 23 at 15:08
See superuser.com/a/1403322/19943 for an image of how they are connected and QPI
– Mokubai♦
Feb 23 at 15:17
add a comment |
There is an inter-processor link between the two CPUs for memory access (see NUMA), I would expect access to shared resources such as GPUs to be handled similarly.
– Mokubai♦
Feb 23 at 14:59
@Mokubai So does they work through memory? Or does they have an additional specialized bus for sharing pci-express functions?
– xakepp35
Feb 23 at 15:08
See superuser.com/a/1403322/19943 for an image of how they are connected and QPI
– Mokubai♦
Feb 23 at 15:17
There is an inter-processor link between the two CPUs for memory access (see NUMA), I would expect access to shared resources such as GPUs to be handled similarly.
– Mokubai♦
Feb 23 at 14:59
There is an inter-processor link between the two CPUs for memory access (see NUMA), I would expect access to shared resources such as GPUs to be handled similarly.
– Mokubai♦
Feb 23 at 14:59
@Mokubai So does they work through memory? Or does they have an additional specialized bus for sharing pci-express functions?
– xakepp35
Feb 23 at 15:08
@Mokubai So does they work through memory? Or does they have an additional specialized bus for sharing pci-express functions?
– xakepp35
Feb 23 at 15:08
See superuser.com/a/1403322/19943 for an image of how they are connected and QPI
– Mokubai♦
Feb 23 at 15:17
See superuser.com/a/1403322/19943 for an image of how they are connected and QPI
– Mokubai♦
Feb 23 at 15:17
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1408776%2fhow-does-multi-cpu-system-shares-pci-express-bus%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1408776%2fhow-does-multi-cpu-system-shares-pci-express-bus%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There is an inter-processor link between the two CPUs for memory access (see NUMA), I would expect access to shared resources such as GPUs to be handled similarly.
– Mokubai♦
Feb 23 at 14:59
@Mokubai So does they work through memory? Or does they have an additional specialized bus for sharing pci-express functions?
– xakepp35
Feb 23 at 15:08
See superuser.com/a/1403322/19943 for an image of how they are connected and QPI
– Mokubai♦
Feb 23 at 15:17