python – pandas SettingWithCopyWarning

DataFrame 의 자료 구조에서  새로운 column을 만들고 column에 값을 넣을 때 아래와 같은 SettingWithCopyWarning 메시지가 발생한다.

<ipython-input-64-04010302304>:1: SettingWithCopyWarning:
A values is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documention: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versius-a-copy

document 문서를 보면 pandas object에 데이터를 넣을 때 chained indexing 를 주의 하라는 메시지가 나오고 example 코드가 나온다.

In [354]: dfmi = pd.DataFrame([list('abcd'),
   .....:                      list('efgh'),
   .....:                      list('ijkl'),
   .....:                      list('mnop')],
   .....:                     columns=pd.MultiIndex.from_product([['one', 'two'],
   .....:                                                         ['first', 'second']]))
   .....: 

In [355]: dfmi
Out[355]: 
    one          two       
  first second first second
0     a      b     c      d
1     e      f     g      h
2     i      j     k      l
3     m      n     o      p
In [356]: dfmi['one']['second']
Out[356]: 
0    b
1    f
2    j
3    n
Name: second, dtype: object
In [357]: dfmi.loc[:, ('one', 'second')]
Out[357]: 
0    b
1    f
2    j
3    n
Name: (one, second), dtype: object

‘one’,’second’의 column의 값을 확인 하기 위한 방법은 첫번째 방법 dfmi[‘one’][‘second’], 두번째 방법 dfmi.loc[:, (‘one’,’second’) 이 있고 같은 결과를 보여준다.

그럼 두 개의 차이점은??

dfmi[‘one’][‘secod’]

우선 dfmi[‘one’]를 선택된 개별적인 index를 가진 dataframe를 리턴을 한다.  그리고 다른 python operation 인 dfmi_with_one[‘second’]은  ‘second’로 인덱싱 된 series를 선택하게 된다. 즉 pandas 는 분리된 이벤트로 __getitem__을 호출하여 데이터를 가져오게 되는데, ‘one’ 의 데이터를 가져온 후, ‘second’을 가져오는 방식으로 선형동작에 문제를 일으킬 수 있다.

dfmi.loc[:,(‘one’,’secod’)]

한번의 __getitem__ 을 호출하여 tuple 집합을 가져온다. pandas는 이 방식을 하나의 객체로 다룬다. 그리고 이러한 연산순서는 속도가 훨씬 빨라 질 수 있고, 두 축을 인덱싱 할 수 있다.

실제로 경고 문구에서 추천하는 방식과 chained indexing 을 통한 동작 방식을 보면 아래와 같다.

dfmi.loc[:, ('one', 'second')] = value
# becomes
dfmi.loc.__setitem__((slice(None), ('one', 'second')), value)
dfmi['one']['second'] = value
# becomes
dfmi.__getitem__('one').__setitem__('second', value)

하지만 chained indexing 을 사용하지 않는 경우에도 경고 문구가 발생할 수 있다. 하지만 SettingWithCopyWarning이 해당 코드의 버그를 주지 시키기 위한 의도된 동작인걸로 보여진다.

그리고 해당 경고에 대한 pandas 옵션을 제공해 주고 있다

pd.set_option('mode.chained_assignment','warn') : default 옵션
pd.set_option('mode.chained_assignment','raise') : SettingWithCopyException 오류를 발생시킨다.
pd.set_option('mode.chained_assignment', None) : 해당 경고 자체를 발생하지 않는다.

그리고 SettingWithCopyWarning 의 문제를 해결하기 위하여 추전하는 방법으로는 아래와 같이 .copy() 를 사용하는 것이다.

In [360]: dfc = pd.DataFrame({'a': ['one', 'one', 'two',
   .....:                           'three', 'two', 'one', 'six'],
   .....:                     'c': np.arange(7)})
   .....: 

In [361]: dfd = dfc.copy()

# Setting multiple items using a mask
In [362]: mask = dfd['a'].str.startswith('o')

In [363]: dfd.loc[mask, 'c'] = 42

In [364]: dfd
Out[364]: 
       a   c
0    one  42
1    one  42
2    two   2
3  three   3
4    two   4
5    one  42
6    six   6

# Setting a single item
In [365]: dfd = dfc.copy()

In [366]: dfd.loc[2, 'a'] = 11

In [367]: dfd
Out[367]: 
       a  c
0    one  0
1    one  1
2     11  2
3  three  3
4    two  4
5    one  5
6    six  6

 

 

“python – pandas SettingWithCopyWarning”의 35개의 댓글

  1. Hi there this is somewhat of off topic but I was wondering if blogs use WYSIWYG
    editors or if you have to manually code with HTML. I’m starting a blog
    soon but have no coding know-how so I wanted to get advice from someone
    with experience. Any help would be enormously appreciated!

  2. Can I simply say what a relief to find somebody
    that actually knows what they’re discussing on the net.
    You actually know how to bring a problem to light and make it important.
    More and more people must look at this and understand
    this side of your story. I was surprised you aren’t more popular since you surely possess the gift.

  3. I got this site from my buddy who informed me concerning this web site and at the moment this time I am browsing this web site and
    reading very informative articles or reviews at this place.

  4. Basically to follow up on the update of this topic on your
    blog and would like to let you know simply how much I appreciated
    the time you took to generate this valuable post. Inside the post, you spoke of how to really handle this matter with
    all comfort. It would be my personal pleasure to
    gather some more ideas from your web-site and come as much as offer
    some others what I have benefited from you. Thank you for your usual wonderful effort.

    Also visit my homepage – SperMax Control Reviews

  5. I’ve been surfing online more than 2 hours today, yet I never found any interesting article like
    yours. It is pretty worth enough for me. In my view, if all webmasters and bloggers made good
    content as you did, the web will be a lot more
    useful than ever before.

  6. Hmm it appears like your blog ate my first comment (it was extremely long)
    so I guess I’ll just sum it up what I submitted and say,
    I’m thoroughly enjoying your blog. I as well am an aspiring blog writer
    but I’m still new to everything. Do you have any helpful hints for
    rookie blog writers? I’d certainly appreciate it.

  7. Have you ever considered about adding a little bit more than just your articles?
    I mean, what you say is valuable and all. Nevertheless imagine if you
    added some great graphics or videos to give your posts more,
    “pop”! Your content is excellent but with images and video clips, this website could certainly be
    one of the greatest in its niche. Excellent blog!

  8. Just want to say your article is as astounding. The clarity in your post is just cool and i could assume you are an expert on this
    subject. Fine with your permission let me to grab your
    feed to keep up to date with forthcoming post.
    Thanks a million and please continue the rewarding work.

  9. Fantastic goods from you, man. I’ve understand
    your stuff previous to and you’re just extremely great.

    I really like what you have acquired here, certainly like what you are stating and the way
    in which you say it. You make it enjoyable and you still care for to keep it wise.
    I can not wait to read much more from you.
    This is actually a tremendous website.

    Also visit my homepage :: Mermber XXL Reviews

  10. I wanted to check up and allow you to know how considerably I
    liked discovering your site today. I’d consider it a real honor
    to work at my company and be able to utilize tips contributed on your site and also engage in visitors’ reviews like this.
    Should a position connected with guest writer become offered at your end, i highly recommend you let me know.

    my website; Viag Rx

  11. You’ve made some really good points there. I checked on the web
    for more info about the issue and found most individuals will go along with your views on this site.

  12. Write more, thats all I have to say. Literally, it seems as though you relied on the video
    to make your point. You definitely know what youre talking about, why
    waste your intelligence on just posting videos to your blog when you
    could be giving us something informative to read?

    Feel free to visit my website :: forum.mkovalcik.cz

  13. I am only commenting to make you know of the Cool Portable AC Reviews discovery my cousin’s
    princess undergone visiting your site. She came to understand many pieces,
    including what it’s like to possess a very effective helping mood to make many people without difficulty
    thoroughly grasp some grueling issues. You actually exceeded readers’ expected results.
    Thanks for rendering those effective, healthy, explanatory and in addition unique thoughts on that topic to Ethel.

  14. Pretty great post. I just stumbled upon your weblog
    and wished to say that I’ve truly loved surfing around your weblog posts.
    After all I will be subscribing in your feed and I am hoping you write once more soon!

    Also visit my blog: Craig

  15. Its such as you learn my mind! You seem to understand so much approximately this, like
    you wrote the e book in it or something. I think that you
    just could do with a few % to pressure the message house a little bit, but instead of that,
    that is great blog. An excellent read. I will definitely be back.

    my homepage; Keto BHB Plus Reviews

답글 남기기