Remove shared references in list-of-list?












17















Ok, let me explain the problem with a simple example:



l = [[0]]*3       # makes the array [[0], [0], [0]]
l[0][0] = 42 # l becomes [[42], [42], [42]]
from copy import deepcopy
m = deepcopy(l) # m becomes [[42], [42], [42]]
m[0][0] = 2 # m becomes [[2], [2], [2]]


This is a basic shared reference problem. Except usually, when a problem like this occurs, deepcopy is our friend.
Currently, I made this to solve my deepcopy betrayal problem:



l = [[0]]*3       # makes the array [[0], [0], [0]]
import JSON
m = JSON.loads(JSON.dumps(l)) # m becomes [[0], [0], [0]] with no self references


I am looking for a less inefficient and less stupid way of handling self shared reference.



Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code. Running my "solution" on large arrays is slow, and I have many levels of nested arrays, I can't afford to make a string this big for those beasts.










share|improve this question




















  • 1





    So the shared reference could be anything? A custom object, a dict? Or just list?

    – Daniel Mesejo
    Jan 31 at 18:45








  • 7





    "but I need to handle the case where someone gives one to my code" - if someone gives you a broken input, their code is broken, and it's their problem. Trying to fix it for them just hides their bug and makes it harder for them to become aware of it. They need to learn to build nested lists correctly.

    – user2357112
    Jan 31 at 18:46






  • 1





    And the depth of the list is arbitrary?

    – Daniel Mesejo
    Jan 31 at 18:48






  • 1





    You can write your own function to copy your list, depending on what exactly the structure you are expecting is and how you want to handle things.

    – juanpa.arrivillaga
    Jan 31 at 18:49






  • 1





    @Idlehands sure it is, you don't make a totally unnecessary string. At the end of the day, you need to create a copy, but at least that would copy things directly. Now, if you really wanted to be efficient, you would have to walk the structure and check for duplicate list objects, and only make a copy of those in-place (hopefully avoiding a lot of necessary copies, but note,for the example, it would be the same)

    – juanpa.arrivillaga
    Jan 31 at 18:52


















17















Ok, let me explain the problem with a simple example:



l = [[0]]*3       # makes the array [[0], [0], [0]]
l[0][0] = 42 # l becomes [[42], [42], [42]]
from copy import deepcopy
m = deepcopy(l) # m becomes [[42], [42], [42]]
m[0][0] = 2 # m becomes [[2], [2], [2]]


This is a basic shared reference problem. Except usually, when a problem like this occurs, deepcopy is our friend.
Currently, I made this to solve my deepcopy betrayal problem:



l = [[0]]*3       # makes the array [[0], [0], [0]]
import JSON
m = JSON.loads(JSON.dumps(l)) # m becomes [[0], [0], [0]] with no self references


I am looking for a less inefficient and less stupid way of handling self shared reference.



Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code. Running my "solution" on large arrays is slow, and I have many levels of nested arrays, I can't afford to make a string this big for those beasts.










share|improve this question




















  • 1





    So the shared reference could be anything? A custom object, a dict? Or just list?

    – Daniel Mesejo
    Jan 31 at 18:45








  • 7





    "but I need to handle the case where someone gives one to my code" - if someone gives you a broken input, their code is broken, and it's their problem. Trying to fix it for them just hides their bug and makes it harder for them to become aware of it. They need to learn to build nested lists correctly.

    – user2357112
    Jan 31 at 18:46






  • 1





    And the depth of the list is arbitrary?

    – Daniel Mesejo
    Jan 31 at 18:48






  • 1





    You can write your own function to copy your list, depending on what exactly the structure you are expecting is and how you want to handle things.

    – juanpa.arrivillaga
    Jan 31 at 18:49






  • 1





    @Idlehands sure it is, you don't make a totally unnecessary string. At the end of the day, you need to create a copy, but at least that would copy things directly. Now, if you really wanted to be efficient, you would have to walk the structure and check for duplicate list objects, and only make a copy of those in-place (hopefully avoiding a lot of necessary copies, but note,for the example, it would be the same)

    – juanpa.arrivillaga
    Jan 31 at 18:52
















17












17








17








Ok, let me explain the problem with a simple example:



l = [[0]]*3       # makes the array [[0], [0], [0]]
l[0][0] = 42 # l becomes [[42], [42], [42]]
from copy import deepcopy
m = deepcopy(l) # m becomes [[42], [42], [42]]
m[0][0] = 2 # m becomes [[2], [2], [2]]


This is a basic shared reference problem. Except usually, when a problem like this occurs, deepcopy is our friend.
Currently, I made this to solve my deepcopy betrayal problem:



l = [[0]]*3       # makes the array [[0], [0], [0]]
import JSON
m = JSON.loads(JSON.dumps(l)) # m becomes [[0], [0], [0]] with no self references


I am looking for a less inefficient and less stupid way of handling self shared reference.



Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code. Running my "solution" on large arrays is slow, and I have many levels of nested arrays, I can't afford to make a string this big for those beasts.










share|improve this question
















Ok, let me explain the problem with a simple example:



l = [[0]]*3       # makes the array [[0], [0], [0]]
l[0][0] = 42 # l becomes [[42], [42], [42]]
from copy import deepcopy
m = deepcopy(l) # m becomes [[42], [42], [42]]
m[0][0] = 2 # m becomes [[2], [2], [2]]


This is a basic shared reference problem. Except usually, when a problem like this occurs, deepcopy is our friend.
Currently, I made this to solve my deepcopy betrayal problem:



l = [[0]]*3       # makes the array [[0], [0], [0]]
import JSON
m = JSON.loads(JSON.dumps(l)) # m becomes [[0], [0], [0]] with no self references


I am looking for a less inefficient and less stupid way of handling self shared reference.



Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code. Running my "solution" on large arrays is slow, and I have many levels of nested arrays, I can't afford to make a string this big for those beasts.







python reference nested-lists deep-copy






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 31 at 19:30









smci

15k676104




15k676104










asked Jan 31 at 18:39









Benoît PilatteBenoît Pilatte

1,361220




1,361220








  • 1





    So the shared reference could be anything? A custom object, a dict? Or just list?

    – Daniel Mesejo
    Jan 31 at 18:45








  • 7





    "but I need to handle the case where someone gives one to my code" - if someone gives you a broken input, their code is broken, and it's their problem. Trying to fix it for them just hides their bug and makes it harder for them to become aware of it. They need to learn to build nested lists correctly.

    – user2357112
    Jan 31 at 18:46






  • 1





    And the depth of the list is arbitrary?

    – Daniel Mesejo
    Jan 31 at 18:48






  • 1





    You can write your own function to copy your list, depending on what exactly the structure you are expecting is and how you want to handle things.

    – juanpa.arrivillaga
    Jan 31 at 18:49






  • 1





    @Idlehands sure it is, you don't make a totally unnecessary string. At the end of the day, you need to create a copy, but at least that would copy things directly. Now, if you really wanted to be efficient, you would have to walk the structure and check for duplicate list objects, and only make a copy of those in-place (hopefully avoiding a lot of necessary copies, but note,for the example, it would be the same)

    – juanpa.arrivillaga
    Jan 31 at 18:52
















  • 1





    So the shared reference could be anything? A custom object, a dict? Or just list?

    – Daniel Mesejo
    Jan 31 at 18:45








  • 7





    "but I need to handle the case where someone gives one to my code" - if someone gives you a broken input, their code is broken, and it's their problem. Trying to fix it for them just hides their bug and makes it harder for them to become aware of it. They need to learn to build nested lists correctly.

    – user2357112
    Jan 31 at 18:46






  • 1





    And the depth of the list is arbitrary?

    – Daniel Mesejo
    Jan 31 at 18:48






  • 1





    You can write your own function to copy your list, depending on what exactly the structure you are expecting is and how you want to handle things.

    – juanpa.arrivillaga
    Jan 31 at 18:49






  • 1





    @Idlehands sure it is, you don't make a totally unnecessary string. At the end of the day, you need to create a copy, but at least that would copy things directly. Now, if you really wanted to be efficient, you would have to walk the structure and check for duplicate list objects, and only make a copy of those in-place (hopefully avoiding a lot of necessary copies, but note,for the example, it would be the same)

    – juanpa.arrivillaga
    Jan 31 at 18:52










1




1





So the shared reference could be anything? A custom object, a dict? Or just list?

– Daniel Mesejo
Jan 31 at 18:45







So the shared reference could be anything? A custom object, a dict? Or just list?

– Daniel Mesejo
Jan 31 at 18:45






7




7





"but I need to handle the case where someone gives one to my code" - if someone gives you a broken input, their code is broken, and it's their problem. Trying to fix it for them just hides their bug and makes it harder for them to become aware of it. They need to learn to build nested lists correctly.

– user2357112
Jan 31 at 18:46





"but I need to handle the case where someone gives one to my code" - if someone gives you a broken input, their code is broken, and it's their problem. Trying to fix it for them just hides their bug and makes it harder for them to become aware of it. They need to learn to build nested lists correctly.

– user2357112
Jan 31 at 18:46




1




1





And the depth of the list is arbitrary?

– Daniel Mesejo
Jan 31 at 18:48





And the depth of the list is arbitrary?

– Daniel Mesejo
Jan 31 at 18:48




1




1





You can write your own function to copy your list, depending on what exactly the structure you are expecting is and how you want to handle things.

– juanpa.arrivillaga
Jan 31 at 18:49





You can write your own function to copy your list, depending on what exactly the structure you are expecting is and how you want to handle things.

– juanpa.arrivillaga
Jan 31 at 18:49




1




1





@Idlehands sure it is, you don't make a totally unnecessary string. At the end of the day, you need to create a copy, but at least that would copy things directly. Now, if you really wanted to be efficient, you would have to walk the structure and check for duplicate list objects, and only make a copy of those in-place (hopefully avoiding a lot of necessary copies, but note,for the example, it would be the same)

– juanpa.arrivillaga
Jan 31 at 18:52







@Idlehands sure it is, you don't make a totally unnecessary string. At the end of the day, you need to create a copy, but at least that would copy things directly. Now, if you really wanted to be efficient, you would have to walk the structure and check for duplicate list objects, and only make a copy of those in-place (hopefully avoiding a lot of necessary copies, but note,for the example, it would be the same)

– juanpa.arrivillaga
Jan 31 at 18:52














7 Answers
7






active

oldest

votes


















10














Here's an approach that will work on any combination of lists, dicts, and immutable values.



def very_deep_copy(obj):
if isinstance(obj, list):
return [very_deep_copy(item) for item in obj]
elif isinstance(obj, dict):
return {k: very_deep_copy(v) for k,v in obj.items()}
else:
return obj

l = [[0]]*3
m = very_deep_copy(l)
m[0][0] = 2
print(m)


Result:



[[2], [0], [0]]





share|improve this answer
























  • but then m isn't an exact copy of l. but I guess this is what OP wants

    – sam46
    Jan 31 at 19:03











  • for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

    – Azat Ibrakov
    Feb 1 at 12:17













  • This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

    – Peilonrayz
    Feb 18 at 16:51





















7














I'm going to challenge the assumption that the right thing to do is to copy the shared objects. You say that




Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code.




but if someone passes you an input with unexpected object sharing, their code has a bug. If your code notices the bug, your code should tell them about it by throwing an exception, to help them fix their bugs.



Most code would just assume the input doesn't have any unwanted object sharing. If you want to detect it anyway, a manual traversal is probably your best option, especially since your input is expected to be JSON-serializable:



def detect_duplicate_references(data):
_detect_duplicate_references(data, set())

def _detect_duplicate_references(data, memo):
if isinstance(data, (dict, list)):
if id(data) in memo:
raise ValueError(f'multiple references to object {data} detected in input')
memo.add(id(data))
if isinstance(data, list):
for obj in data:
_detect_duplicate_references(obj, memo)
if isinstance(data, dict):
for obj in data.values():
_detect_duplicate_references(obj, memo)





share|improve this answer



















  • 2





    I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

    – Idlehands
    Jan 31 at 19:13





















2














l = [[0] for _ in range(3)]
l[0][0] = 2
print(l)


prints:



[[2], [0], [0]]


also, btw in your code deepcopy() gave the output it did because you passed in a list which already had elements that share the same reference.



from copy import deepcopy
m = deepcopy(l)
m[0][0] = 3
print(l) # prints [[2], [0], [0]]
print(m) # prints [[3], [0], [0]]


you can see here it does what we expect it to no problem






share|improve this answer





















  • 1





    I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

    – Rocky Li
    Jan 31 at 18:45



















1














Assuming only structure type will be list, this should work for list of arbitrary depth and complexity:



def customcopy(l):
return [customcopy(e) if type(e) is list else e for e in l]

l = [[0]]*3
x = customcopy(l)
x[0][0] = 3

>>> x
[[3], [0], [0]]





share|improve this answer



















  • 1





    OP already mentioned it is a mix lists and dicts.

    – AChampion
    Jan 31 at 18:57











  • Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

    – Benoît Pilatte
    Jan 31 at 18:59











  • @Kevin made the same solution work for combinations of dict and list

    – Benoît Pilatte
    Jan 31 at 19:04



















0














Not sure it's efficient but you could try:



l = [deepcopy(elt) for elt in l]





share|improve this answer































    0














    For a one-level-deep copy, you can just check the references. For deeper copies, just do this recursively.



    from copy import deepcopy

    def copy_value(l):
    l = list(l)
    new_list =
    while l:
    item = l.pop(0)
    if item not in new_list:
    new_list.append(item)
    else:
    new_list.append(deepcopy(item))
    return new_list

    l = [[0]]*3
    m = copy_value(l)
    m[0][0] = 2
    print(m)


    prints



    [[2], [0], [0]]





    share|improve this answer































      0














      Another way with a list comprehension:



      def super_deep_copy(l):
      return [i.copy() for i in l]
      l = [[0]]*3
      l = super_deep_copy(l)
      l[0][0] = 2


      And now:



      print(l)


      Is:



      [[2], [0], [0]]





      share|improve this answer

























        Your Answer






        StackExchange.ifUsing("editor", function () {
        StackExchange.using("externalEditor", function () {
        StackExchange.using("snippets", function () {
        StackExchange.snippets.init();
        });
        });
        }, "code-snippets");

        StackExchange.ready(function() {
        var channelOptions = {
        tags: "".split(" "),
        id: "1"
        };
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function() {
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled) {
        StackExchange.using("snippets", function() {
        createEditor();
        });
        }
        else {
        createEditor();
        }
        });

        function createEditor() {
        StackExchange.prepareEditor({
        heartbeatType: 'answer',
        autoActivateHeartbeat: false,
        convertImagesToLinks: true,
        noModals: true,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: 10,
        bindNavPrevention: true,
        postfix: "",
        imageUploader: {
        brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
        contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
        allowUrls: true
        },
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        });


        }
        });














        draft saved

        draft discarded


















        StackExchange.ready(
        function () {
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54467214%2fremove-shared-references-in-list-of-list%23new-answer', 'question_page');
        }
        );

        Post as a guest















        Required, but never shown

























        7 Answers
        7






        active

        oldest

        votes








        7 Answers
        7






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes









        10














        Here's an approach that will work on any combination of lists, dicts, and immutable values.



        def very_deep_copy(obj):
        if isinstance(obj, list):
        return [very_deep_copy(item) for item in obj]
        elif isinstance(obj, dict):
        return {k: very_deep_copy(v) for k,v in obj.items()}
        else:
        return obj

        l = [[0]]*3
        m = very_deep_copy(l)
        m[0][0] = 2
        print(m)


        Result:



        [[2], [0], [0]]





        share|improve this answer
























        • but then m isn't an exact copy of l. but I guess this is what OP wants

          – sam46
          Jan 31 at 19:03











        • for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

          – Azat Ibrakov
          Feb 1 at 12:17













        • This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

          – Peilonrayz
          Feb 18 at 16:51


















        10














        Here's an approach that will work on any combination of lists, dicts, and immutable values.



        def very_deep_copy(obj):
        if isinstance(obj, list):
        return [very_deep_copy(item) for item in obj]
        elif isinstance(obj, dict):
        return {k: very_deep_copy(v) for k,v in obj.items()}
        else:
        return obj

        l = [[0]]*3
        m = very_deep_copy(l)
        m[0][0] = 2
        print(m)


        Result:



        [[2], [0], [0]]





        share|improve this answer
























        • but then m isn't an exact copy of l. but I guess this is what OP wants

          – sam46
          Jan 31 at 19:03











        • for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

          – Azat Ibrakov
          Feb 1 at 12:17













        • This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

          – Peilonrayz
          Feb 18 at 16:51
















        10












        10








        10







        Here's an approach that will work on any combination of lists, dicts, and immutable values.



        def very_deep_copy(obj):
        if isinstance(obj, list):
        return [very_deep_copy(item) for item in obj]
        elif isinstance(obj, dict):
        return {k: very_deep_copy(v) for k,v in obj.items()}
        else:
        return obj

        l = [[0]]*3
        m = very_deep_copy(l)
        m[0][0] = 2
        print(m)


        Result:



        [[2], [0], [0]]





        share|improve this answer













        Here's an approach that will work on any combination of lists, dicts, and immutable values.



        def very_deep_copy(obj):
        if isinstance(obj, list):
        return [very_deep_copy(item) for item in obj]
        elif isinstance(obj, dict):
        return {k: very_deep_copy(v) for k,v in obj.items()}
        else:
        return obj

        l = [[0]]*3
        m = very_deep_copy(l)
        m[0][0] = 2
        print(m)


        Result:



        [[2], [0], [0]]






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 31 at 18:50









        KevinKevin

        58.7k1169113




        58.7k1169113













        • but then m isn't an exact copy of l. but I guess this is what OP wants

          – sam46
          Jan 31 at 19:03











        • for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

          – Azat Ibrakov
          Feb 1 at 12:17













        • This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

          – Peilonrayz
          Feb 18 at 16:51





















        • but then m isn't an exact copy of l. but I guess this is what OP wants

          – sam46
          Jan 31 at 19:03











        • for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

          – Azat Ibrakov
          Feb 1 at 12:17













        • This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

          – Peilonrayz
          Feb 18 at 16:51



















        but then m isn't an exact copy of l. but I guess this is what OP wants

        – sam46
        Jan 31 at 19:03





        but then m isn't an exact copy of l. but I guess this is what OP wants

        – sam46
        Jan 31 at 19:03













        for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

        – Azat Ibrakov
        Feb 1 at 12:17







        for further readers: this solution will fail on recursive structures like x = ; x.append(x) (which not so common, but we should bear them in mind) with RecursionError

        – Azat Ibrakov
        Feb 1 at 12:17















        This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

        – Peilonrayz
        Feb 18 at 16:51







        This, like the question, doesn't work with custom Sequences. Using EAFP rather than LBYL would allow it to. Causing you to lose the original type, which could be brought back with a little more effort.

        – Peilonrayz
        Feb 18 at 16:51















        7














        I'm going to challenge the assumption that the right thing to do is to copy the shared objects. You say that




        Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code.




        but if someone passes you an input with unexpected object sharing, their code has a bug. If your code notices the bug, your code should tell them about it by throwing an exception, to help them fix their bugs.



        Most code would just assume the input doesn't have any unwanted object sharing. If you want to detect it anyway, a manual traversal is probably your best option, especially since your input is expected to be JSON-serializable:



        def detect_duplicate_references(data):
        _detect_duplicate_references(data, set())

        def _detect_duplicate_references(data, memo):
        if isinstance(data, (dict, list)):
        if id(data) in memo:
        raise ValueError(f'multiple references to object {data} detected in input')
        memo.add(id(data))
        if isinstance(data, list):
        for obj in data:
        _detect_duplicate_references(obj, memo)
        if isinstance(data, dict):
        for obj in data.values():
        _detect_duplicate_references(obj, memo)





        share|improve this answer



















        • 2





          I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

          – Idlehands
          Jan 31 at 19:13


















        7














        I'm going to challenge the assumption that the right thing to do is to copy the shared objects. You say that




        Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code.




        but if someone passes you an input with unexpected object sharing, their code has a bug. If your code notices the bug, your code should tell them about it by throwing an exception, to help them fix their bugs.



        Most code would just assume the input doesn't have any unwanted object sharing. If you want to detect it anyway, a manual traversal is probably your best option, especially since your input is expected to be JSON-serializable:



        def detect_duplicate_references(data):
        _detect_duplicate_references(data, set())

        def _detect_duplicate_references(data, memo):
        if isinstance(data, (dict, list)):
        if id(data) in memo:
        raise ValueError(f'multiple references to object {data} detected in input')
        memo.add(id(data))
        if isinstance(data, list):
        for obj in data:
        _detect_duplicate_references(obj, memo)
        if isinstance(data, dict):
        for obj in data.values():
        _detect_duplicate_references(obj, memo)





        share|improve this answer



















        • 2





          I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

          – Idlehands
          Jan 31 at 19:13
















        7












        7








        7







        I'm going to challenge the assumption that the right thing to do is to copy the shared objects. You say that




        Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code.




        but if someone passes you an input with unexpected object sharing, their code has a bug. If your code notices the bug, your code should tell them about it by throwing an exception, to help them fix their bugs.



        Most code would just assume the input doesn't have any unwanted object sharing. If you want to detect it anyway, a manual traversal is probably your best option, especially since your input is expected to be JSON-serializable:



        def detect_duplicate_references(data):
        _detect_duplicate_references(data, set())

        def _detect_duplicate_references(data, memo):
        if isinstance(data, (dict, list)):
        if id(data) in memo:
        raise ValueError(f'multiple references to object {data} detected in input')
        memo.add(id(data))
        if isinstance(data, list):
        for obj in data:
        _detect_duplicate_references(obj, memo)
        if isinstance(data, dict):
        for obj in data.values():
        _detect_duplicate_references(obj, memo)





        share|improve this answer













        I'm going to challenge the assumption that the right thing to do is to copy the shared objects. You say that




        Of course I wouldn't make arrays like that on purpose, but I need to handle the case where someone gives one to my code.




        but if someone passes you an input with unexpected object sharing, their code has a bug. If your code notices the bug, your code should tell them about it by throwing an exception, to help them fix their bugs.



        Most code would just assume the input doesn't have any unwanted object sharing. If you want to detect it anyway, a manual traversal is probably your best option, especially since your input is expected to be JSON-serializable:



        def detect_duplicate_references(data):
        _detect_duplicate_references(data, set())

        def _detect_duplicate_references(data, memo):
        if isinstance(data, (dict, list)):
        if id(data) in memo:
        raise ValueError(f'multiple references to object {data} detected in input')
        memo.add(id(data))
        if isinstance(data, list):
        for obj in data:
        _detect_duplicate_references(obj, memo)
        if isinstance(data, dict):
        for obj in data.values():
        _detect_duplicate_references(obj, memo)






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 31 at 19:01









        user2357112user2357112

        155k12167260




        155k12167260








        • 2





          I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

          – Idlehands
          Jan 31 at 19:13
















        • 2





          I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

          – Idlehands
          Jan 31 at 19:13










        2




        2





        I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

        – Idlehands
        Jan 31 at 19:13







        I like this. +1 This could easily be used to create a deep copy like @Kevin's answer but it's gloriously pedantic and feels so right.

        – Idlehands
        Jan 31 at 19:13













        2














        l = [[0] for _ in range(3)]
        l[0][0] = 2
        print(l)


        prints:



        [[2], [0], [0]]


        also, btw in your code deepcopy() gave the output it did because you passed in a list which already had elements that share the same reference.



        from copy import deepcopy
        m = deepcopy(l)
        m[0][0] = 3
        print(l) # prints [[2], [0], [0]]
        print(m) # prints [[3], [0], [0]]


        you can see here it does what we expect it to no problem






        share|improve this answer





















        • 1





          I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

          – Rocky Li
          Jan 31 at 18:45
















        2














        l = [[0] for _ in range(3)]
        l[0][0] = 2
        print(l)


        prints:



        [[2], [0], [0]]


        also, btw in your code deepcopy() gave the output it did because you passed in a list which already had elements that share the same reference.



        from copy import deepcopy
        m = deepcopy(l)
        m[0][0] = 3
        print(l) # prints [[2], [0], [0]]
        print(m) # prints [[3], [0], [0]]


        you can see here it does what we expect it to no problem






        share|improve this answer





















        • 1





          I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

          – Rocky Li
          Jan 31 at 18:45














        2












        2








        2







        l = [[0] for _ in range(3)]
        l[0][0] = 2
        print(l)


        prints:



        [[2], [0], [0]]


        also, btw in your code deepcopy() gave the output it did because you passed in a list which already had elements that share the same reference.



        from copy import deepcopy
        m = deepcopy(l)
        m[0][0] = 3
        print(l) # prints [[2], [0], [0]]
        print(m) # prints [[3], [0], [0]]


        you can see here it does what we expect it to no problem






        share|improve this answer















        l = [[0] for _ in range(3)]
        l[0][0] = 2
        print(l)


        prints:



        [[2], [0], [0]]


        also, btw in your code deepcopy() gave the output it did because you passed in a list which already had elements that share the same reference.



        from copy import deepcopy
        m = deepcopy(l)
        m[0][0] = 3
        print(l) # prints [[2], [0], [0]]
        print(m) # prints [[3], [0], [0]]


        you can see here it does what we expect it to no problem







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Jan 31 at 18:54

























        answered Jan 31 at 18:42









        sam46sam46

        1,058711




        1,058711








        • 1





          I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

          – Rocky Li
          Jan 31 at 18:45














        • 1





          I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

          – Rocky Li
          Jan 31 at 18:45








        1




        1





        I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

        – Rocky Li
        Jan 31 at 18:45





        I don't think OP is asking for ways to avoid a deep copy, but rather ways to get out of a deep copy.

        – Rocky Li
        Jan 31 at 18:45











        1














        Assuming only structure type will be list, this should work for list of arbitrary depth and complexity:



        def customcopy(l):
        return [customcopy(e) if type(e) is list else e for e in l]

        l = [[0]]*3
        x = customcopy(l)
        x[0][0] = 3

        >>> x
        [[3], [0], [0]]





        share|improve this answer



















        • 1





          OP already mentioned it is a mix lists and dicts.

          – AChampion
          Jan 31 at 18:57











        • Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

          – Benoît Pilatte
          Jan 31 at 18:59











        • @Kevin made the same solution work for combinations of dict and list

          – Benoît Pilatte
          Jan 31 at 19:04
















        1














        Assuming only structure type will be list, this should work for list of arbitrary depth and complexity:



        def customcopy(l):
        return [customcopy(e) if type(e) is list else e for e in l]

        l = [[0]]*3
        x = customcopy(l)
        x[0][0] = 3

        >>> x
        [[3], [0], [0]]





        share|improve this answer



















        • 1





          OP already mentioned it is a mix lists and dicts.

          – AChampion
          Jan 31 at 18:57











        • Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

          – Benoît Pilatte
          Jan 31 at 18:59











        • @Kevin made the same solution work for combinations of dict and list

          – Benoît Pilatte
          Jan 31 at 19:04














        1












        1








        1







        Assuming only structure type will be list, this should work for list of arbitrary depth and complexity:



        def customcopy(l):
        return [customcopy(e) if type(e) is list else e for e in l]

        l = [[0]]*3
        x = customcopy(l)
        x[0][0] = 3

        >>> x
        [[3], [0], [0]]





        share|improve this answer













        Assuming only structure type will be list, this should work for list of arbitrary depth and complexity:



        def customcopy(l):
        return [customcopy(e) if type(e) is list else e for e in l]

        l = [[0]]*3
        x = customcopy(l)
        x[0][0] = 3

        >>> x
        [[3], [0], [0]]






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 31 at 18:54









        Rocky LiRocky Li

        3,2311516




        3,2311516








        • 1





          OP already mentioned it is a mix lists and dicts.

          – AChampion
          Jan 31 at 18:57











        • Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

          – Benoît Pilatte
          Jan 31 at 18:59











        • @Kevin made the same solution work for combinations of dict and list

          – Benoît Pilatte
          Jan 31 at 19:04














        • 1





          OP already mentioned it is a mix lists and dicts.

          – AChampion
          Jan 31 at 18:57











        • Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

          – Benoît Pilatte
          Jan 31 at 18:59











        • @Kevin made the same solution work for combinations of dict and list

          – Benoît Pilatte
          Jan 31 at 19:04








        1




        1





        OP already mentioned it is a mix lists and dicts.

        – AChampion
        Jan 31 at 18:57





        OP already mentioned it is a mix lists and dicts.

        – AChampion
        Jan 31 at 18:57













        Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

        – Benoît Pilatte
        Jan 31 at 18:59





        Yes I did, but I like this solution, I think it will be possible to make it compatible with dictionaries.

        – Benoît Pilatte
        Jan 31 at 18:59













        @Kevin made the same solution work for combinations of dict and list

        – Benoît Pilatte
        Jan 31 at 19:04





        @Kevin made the same solution work for combinations of dict and list

        – Benoît Pilatte
        Jan 31 at 19:04











        0














        Not sure it's efficient but you could try:



        l = [deepcopy(elt) for elt in l]





        share|improve this answer




























          0














          Not sure it's efficient but you could try:



          l = [deepcopy(elt) for elt in l]





          share|improve this answer


























            0












            0








            0







            Not sure it's efficient but you could try:



            l = [deepcopy(elt) for elt in l]





            share|improve this answer













            Not sure it's efficient but you could try:



            l = [deepcopy(elt) for elt in l]






            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Jan 31 at 18:44









            Pierre-Nicolas PiquinPierre-Nicolas Piquin

            4668




            4668























                0














                For a one-level-deep copy, you can just check the references. For deeper copies, just do this recursively.



                from copy import deepcopy

                def copy_value(l):
                l = list(l)
                new_list =
                while l:
                item = l.pop(0)
                if item not in new_list:
                new_list.append(item)
                else:
                new_list.append(deepcopy(item))
                return new_list

                l = [[0]]*3
                m = copy_value(l)
                m[0][0] = 2
                print(m)


                prints



                [[2], [0], [0]]





                share|improve this answer




























                  0














                  For a one-level-deep copy, you can just check the references. For deeper copies, just do this recursively.



                  from copy import deepcopy

                  def copy_value(l):
                  l = list(l)
                  new_list =
                  while l:
                  item = l.pop(0)
                  if item not in new_list:
                  new_list.append(item)
                  else:
                  new_list.append(deepcopy(item))
                  return new_list

                  l = [[0]]*3
                  m = copy_value(l)
                  m[0][0] = 2
                  print(m)


                  prints



                  [[2], [0], [0]]





                  share|improve this answer


























                    0












                    0








                    0







                    For a one-level-deep copy, you can just check the references. For deeper copies, just do this recursively.



                    from copy import deepcopy

                    def copy_value(l):
                    l = list(l)
                    new_list =
                    while l:
                    item = l.pop(0)
                    if item not in new_list:
                    new_list.append(item)
                    else:
                    new_list.append(deepcopy(item))
                    return new_list

                    l = [[0]]*3
                    m = copy_value(l)
                    m[0][0] = 2
                    print(m)


                    prints



                    [[2], [0], [0]]





                    share|improve this answer













                    For a one-level-deep copy, you can just check the references. For deeper copies, just do this recursively.



                    from copy import deepcopy

                    def copy_value(l):
                    l = list(l)
                    new_list =
                    while l:
                    item = l.pop(0)
                    if item not in new_list:
                    new_list.append(item)
                    else:
                    new_list.append(deepcopy(item))
                    return new_list

                    l = [[0]]*3
                    m = copy_value(l)
                    m[0][0] = 2
                    print(m)


                    prints



                    [[2], [0], [0]]






                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Jan 31 at 19:06









                    TimTim

                    42129




                    42129























                        0














                        Another way with a list comprehension:



                        def super_deep_copy(l):
                        return [i.copy() for i in l]
                        l = [[0]]*3
                        l = super_deep_copy(l)
                        l[0][0] = 2


                        And now:



                        print(l)


                        Is:



                        [[2], [0], [0]]





                        share|improve this answer






























                          0














                          Another way with a list comprehension:



                          def super_deep_copy(l):
                          return [i.copy() for i in l]
                          l = [[0]]*3
                          l = super_deep_copy(l)
                          l[0][0] = 2


                          And now:



                          print(l)


                          Is:



                          [[2], [0], [0]]





                          share|improve this answer




























                            0












                            0








                            0







                            Another way with a list comprehension:



                            def super_deep_copy(l):
                            return [i.copy() for i in l]
                            l = [[0]]*3
                            l = super_deep_copy(l)
                            l[0][0] = 2


                            And now:



                            print(l)


                            Is:



                            [[2], [0], [0]]





                            share|improve this answer















                            Another way with a list comprehension:



                            def super_deep_copy(l):
                            return [i.copy() for i in l]
                            l = [[0]]*3
                            l = super_deep_copy(l)
                            l[0][0] = 2


                            And now:



                            print(l)


                            Is:



                            [[2], [0], [0]]






                            share|improve this answer














                            share|improve this answer



                            share|improve this answer








                            edited Feb 1 at 5:31

























                            answered Feb 1 at 2:39









                            U9-ForwardU9-Forward

                            15.7k51541




                            15.7k51541






























                                draft saved

                                draft discarded




















































                                Thanks for contributing an answer to Stack Overflow!


                                • Please be sure to answer the question. Provide details and share your research!

                                But avoid



                                • Asking for help, clarification, or responding to other answers.

                                • Making statements based on opinion; back them up with references or personal experience.


                                To learn more, see our tips on writing great answers.




                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function () {
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54467214%2fremove-shared-references-in-list-of-list%23new-answer', 'question_page');
                                }
                                );

                                Post as a guest















                                Required, but never shown





















































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown

































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown







                                Popular posts from this blog

                                Probability when a professor distributes a quiz and homework assignment to a class of n students.

                                Aardman Animations

                                Are they similar matrix