This immediate interactivity is sorely felt when it is missing.

Displaying textual content in Jupyter notebooks should be fast

Once I was going through code submitted by participants for a hackathon for archival purposes. I didn’t need to run any notebooks; just needed to skim through them as part of the process. There was one particular notebook that was full of graphs, generated by a couple of loops. Did you know that images in code cell outputs are stored as base64 encoded strings in the notebook itself? This particular notebook weighted in at >60 MBs, and it froze my Jupyter Lab for a minute or two upon opening it.

Although I don’t run into monster notebooks often, I run into similar productivity snags in my day to day frequent enough that I think about it. Sometimes I just want to quickly skim a notebook to refresh my memory / to look for something I forgot without intending to run the notebook¹.

Jupyter notebooks tend to take a couple of seconds to load in Jupyter Lab. It would be nice if I can flip through content-heavy Jupyter notebooks with the same responsiveness of flipping through pages on a physical notebook. Jupyter notebooks are just specially formatted JSON! I shouldn’t need to start a Jupyter server if all I wanted to do is skim it quickly. Reading and displaying a Jupyter notebook JSON should be fast if we just opt to put plain text² in front of eyeballs.

Why the command line?

I ended up as a heavy command line user after getting used to doing everything on my work server over SSH. Plus, I already use charmbracelet/glow to render markdown in CLI, something similar will work just fine for me.

Low hanging fruit: someone has implemented it!

After studying the format of a Jupyter notebook (link to nbformat docs), I set out to find libraries for building TUI apps in Python (most comfortable language). I hit the jackpot when I came across textualize/rich-cli, which straight up can pretty print notebooks to terminal! It was as simple as pip install rich-cli, then rich notebook.ipynb.

Tried it out, love how it looks. But this story has not come to an end just yet. When loading large notebooks, there was a perceptible lag of a few seconds before the rendered output shows up. Since we’re just rendering text here, I figured there’s a bottleneck somewhere that can be removed to make the output show up almost instantly.

A minor tweak to fully scratch my itch

After some tinkering, I found that the lag came from rich-cli preparing all notebook cells for rendering before dumping them to STDOUT all at once. Modifying the logic to render each cell directly before processing the next one is doable, but this breaks the built-in pagination: the pager in the underlying API (textualize/rich) will display each cell in their own pager instance if I do that. As a workaround, I used subprocess to pipe to less as a pager instead. Because of how bash pipe works, any output received is immediately flushed to STDOUT. Thus, regardless of notebook size, the first few cells are rendered in terminal almost instantly. Bingo!

Now that the logic works as intended, I packaged the code as nbread and put it on github (tnwei/nbread), so that I can easily install it as a standalone command-line application with pipx.

From good to awesome: integration with ranger file manager

I happen to be a happy user of ranger, a command-line file manager that comes with a file preview pane. By default, Jupyter notebooks show up simply as raw JSON in the preview, and is opened as a text file when selected. Given the work done above, I saw how I can improve my user experience by configuring integration with nbread: I won’t even need to launch nbread manually; I can just open the notebook I want by finger-dancing on arrow keys!

Just had to make the following changes:

Modify handle_extension() in ~/.config/ranger/scope.sh for file preview:

case "$extension" in
    ipynb)
    ¦   # Jupyter notebook previewer
    ¦   nbread "${FILE_PATH}" && { dump | trim; exit 5; } || exit 2;;

And adding the following to ~/.config/ranger/rifle.conf to launch nbread when a Jupyter notebook is selected:

# Jupyter notebooks
ext ipynb = nbread "$1" --pager

Conclusion and thoughts

I ended up using the outcome of this little side project almost every day. It just works and doesn’t take much conscious thought to use. Here’s what using nbread in ranger looks like:

Certainly, there are limitations: none of the rich media content like images and interactive widgets are visible (see footnote 2) . However, as a preview tool, it works more than well enough for my purposes and successfully scratched my itch. So I’m happy with it.

(Original blogpost: Read Jupyter notebooks from the command line)

1. *GPT it? Hmmm
2. What about the rich media formatting, you ask? One, typically I’m looking for textual content instead of rich content when skimming through notebooks: code or notes written in comments / markdown cells. And two, dealing with rich content will make this little side project more complex by a few orders of magnitude. I personally don’t think putting a lot of effort here is worth it.

We hosted the first Geohack a few years ago in 2019, as a side dish to the APGCE 2019 conference. Ten teams of five came together to build solutions within the intersection of machine learning and exploration geoscience. As APGCE 2022 will be held physically in Kuala Lumpur again this year, we jumped at the chance to organize Geohack once more.

As we count down one month to the event, details are being hammered out like steel in a forge. Although we are currently in the thick of things, I thought it would be a good idea to take advantage of present-day clarity and document our efforts to serve as reference for posterity. In this post, we’ll talk about how we intend to set up the heart of the hackathon: the data, the challenges, and scoring.

Disclaimer: if you are a prospective participant, take all this information with a grain of salt; things are subject to change as they are.

Data selection

We want the hackathon to be unique. Thus, using data from active Malaysian oilfields in production came to mind. How many subsurface hackathons in the world let you work on data that still has commercial value?

As great as that sounds, permission to use this live data comes with strict stipulations. In order to comply with Malaysian data export regulations, we would need to ensure participants can only access the data during the event itself. But that’s just the easy part. The hard part is to ensure the participants no longer have access to the data after the event. Although it is possible to monitor data transfer throughout the event (don’t ask how), the team isn’t too keen to do that. We would know; we’ve done this in the last Geohack.

The alternative would be to use publicly available oil and gas datasets. Our lives would indeed be easier, but with the scant few datasets available out there, we have the impression that these datasets will already have been utilized for some hackathon or project one way or another.

Eventually, the thought came to us: why not both? Kaggle, the online machine learning competition platform, runs a dual scoring system where submitted models are scored on a public and private leaderboard. The public leaderboard is based on public validation data that is accessible by participants, while the private leaderboard (which decides the final ranking) is based on private validation data that is not shown to participants. We plan to emulate the concept in spirit: repurpose existing public oil and gas datasets for the hackathon, while allowing participants to run their final solution on the Malaysian field data in a controlled environment. We plan to use Dataiku to serve as a virtual data room of sorts.

We’re still tinkering with the interplay between public and private data when it comes to the challenges. Speaking of which …

Challenges

For challenge design, we decided to use a bottom-up approach by listing all possible challenges we can recall / think of, then zoomed in on a select few as a group. At the moment, we narrowed it down to two well-specific challenges and two seismic-specific challenges. We made a conscious decision to limit the number of challenges for two reasons: (i) to focus our data prep efforts better, and (ii) too many challenges makes it difficult to balance them for scoring. We hope that the challenges will be able to cater to teams of varying skillsets; ultimately we just want everyone to have fun at Geohack.

Speaking of scoring …

Scoring

I know we mentioned Kaggle earlier, but we don’t plan to score the hackathon like a Kaggle competition. We want Geohack to be open and welcoming to everybody regardless of skill level; making it a score race will defeat the purpose of the event. Instead, participants will be asked to present their solutions via pitching, preferably with a demo. The scoring criteria will take into account creativity, execution, impact, and presentation. No surprises here; our scoring will resemble a standard hackathon.

Wrapping up

The above information is as much as we can reveal at this point in time. We’ll share more as the events approaches; hope you enjoyed the peek behind the scenes! If you are interested, the event page and signup link can be found at https://icep.com.my/apgce/geohack.

• R² is the coefficient of determination, a measure of how well is the data explained by the fitted model,
• R² is the square of the coefficient of correlation, R,
• R is a quantity that ranges from 0 to 1

Therefore, R² should also range from 0 to 1.

Colour me surprised when the `r2_score` implementation in sklearn returned negative scores. What gives?

This article is adapted from my original blogpost here

The answer lies in the definition

R² is defined upon the basis that the total sum of squares of a fitted model is equal to the explained sum of squares plus the residual sum of squares, or:

where:

• Total sum of squares (SS_tot​) represent the total variation in data, measured by the sum of squares of the difference between expected and actual values,
• Explained sum of squares (SS_exp​) represent the variation in data explained by the fitted model, and
• Residual sum of squares (SS_res​) represent variation in data that is not explained by the fitted model.

R² itself is defined as follows:

Given these definitions, note that negative R² is only possible when the residual sum of squares (SS_res) exceeds the total sum of squares (SS_tot). As this is not mathematically possible, it can only mean that the explained sum of squares and residual sum of squares no longer add up to equal the total sum of squares. In other words, the equality in Equation 1 does not appear [1] to be true.

How can this be?

Because we evaluate models separately on train and test data

Following the above definitions, SS_tot can be calculated using just the data itself, while SS_res depends both on model predictions and the data. While we can use any arbitrary model to generate the predictions for scoring, we need to realize that the aforementioned equality is defined for models trained on the same data. Therefore, it doesn’t necessarily hold true when we use test data to evaluate models built on train data! There is no guarantee that the differences between a foreign model’s predictions and the data is smaller than the variation within the data itself.

We can demonstrate this empirically. The code below fits a couple of linear regression models on randomly generated data:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
import numpy as np

for _ in range(20):
    data = np.random.normal(size=(200, 10))

    X_train = data[:160, :-1]
    X_test = data[160:, :-1]
    y_train = data[:160, -1]
    y_test = data[160:, -1]

    lr = LinearRegression()
    lr.fit(X_train, y_train)

    y_train_pred = lr.predict(X_train)
    y_test_pred = lr.predict(X_test)

    train_score = r2_score(y_train, y_train_pred)
    test_score = r2_score(y_test, y_test_pred)
    print(f"Train R2: {train_score:.3f}, Test R2: {test_score:.3f}")

Try as we might, the R² never drops below zero when the models are evaluated on train data. Here’s what I got in STDOUT:

Train R2: 0.079, Test R2: -0.059
Train R2: 0.019, Test R2: -0.046
Train R2: 0.084, Test R2: -0.060
Train R2: 0.020, Test R2: -0.083
Train R2: 0.065, Test R2: -0.145
Train R2: 0.022, Test R2: 0.032
Train R2: 0.048, Test R2: 0.107
Train R2: 0.076, Test R2: -0.031
Train R2: 0.029, Test R2: 0.006
Train R2: 0.069, Test R2: -0.150
Train R2: 0.064, Test R2: -0.150
Train R2: 0.053, Test R2: 0.096
Train R2: 0.062, Test R2: 0.022
Train R2: 0.063, Test R2: 0.008
Train R2: 0.059, Test R2: -0.061
Train R2: 0.076, Test R2: -0.191
Train R2: 0.049, Test R2: 0.099
Train R2: 0.040, Test R2: -0.012
Train R2: 0.096, Test R2: -0.373
Train R2: 0.073, Test R2: 0.088

So … what about R² being the square of correlation?

It appears that R² = R * R only under limited circumstances. Quoting the paragraph below from the relevant Wikipedia page:

There are several definitions of R² that are only sometimes equivalent. One class of such cases includes that of simple linear regression where r² is used instead of R². When only an intercept is included, then r² is simply the square of the sample correlation coefficient (i.e., r) between the observed outcomes and the observed predictor values. If additional regressors are included, R² is the square of the coefficient of multiple correlation. In both such cases, the coefficient of determination normally ranges from 0 to 1.

In short, R² is only the square of correlation if we happen to be (1) using linear regression models, and (2) are evaluating them on the same data they are fitted (as established previously).

On the liberal use of R² outside the context of linear regression

The quoted Wikipedia paragraph lines up with my observation flipping through statistical texts: R² is almost always introduced within the context of linear regression. That being said, the formulation of R² makes it universally defined for any arbitrary predictive model, regardless of statistical basis. It is used liberally by data scientists in regression tasks, and is even the default metric for regression models in sklearn. Is it right for us to use R² so freely outside its original context?

Honestly, I don’t know. On one hand, it clearly has a lot of utility as a metric, which led to its widespread adoption by data scientists in the first place. On the other hand, you can find discussions like these online that caution against using R² for non-linear regression. It does seem to me that from a statistics perspective, it is important for R² to be calculated under the right conditions such that its properties can be utilized for further analysis. I take my observed relative lack of discourse about R within data science circles to mean that from a data science perspective, R² doesn’t mean more than being a performance metric like MSE or MAE.

Personally, I think we are good with using R², as long as we understand it enough to know what not to do with it.

Wrapping up

To summarize, we should expect R² to be bounded between zero and one only if a linear regression model is fit, and it is evaluated on the same data it is fitted on. Else, the definition of R² can lead to negative values.

[1] Being specific with my choice of words here :)

Explaining negative R-squared was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

(tl;dr: jump straight to Getting rid of SettingWithCopyWarnings if you’re here for answers)

If you are a pandas user, chances are you’ve seen the SettingWithCopyWarning crop up when you’re assigning values to a pd.DataFrame or pd.Series.

In [1]: import pandas as pd
     …:
     …: df = pd.DataFrame({
     …:     “A”: [1, 2, 3, 4, 5],
     …:     “B”: [6, 7, 8, 9, 10],
     …:     }, index=range(5)
     …: )
     …: dfa = df.loc[3:5]
     …: dfa[“C”] = dfa[“B”] * 50

<ipython-input-2–63497d1da3d9>:9: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfa[“C”] = dfa[“B”] * 50

pandas drops a warning here because the value assignment might or might not have worked as expected.

To be clear, the value assignment did occur all right; the emphasis here is on “expected”.

In [2]: dfa
Out[2]:
  A B C
3 4 9 450
4 5 10 500

In [3]: df
Out[3]:
  A B
0 1 6
1 2 7
2 3 8
3 4 9
4 5 10

Did you expect the contents of dfto be affected by the value assignment in dfa? pandas has internally consistent (albeit obtuse) rules on whether that happens or not. It’s just that the ambiguity in user expectations present in this situation warrants a warning, so that end users like you and me know where to look when our code misbehaves.

Chained assignment with views and copies

The act of selecting rows or columns to access from a dataframe or series is called indexing. The flexibility of pandas allows for chained indexing, where you can repeatedly index the outcome of a previous indexing operation.

# Select the 2nd to 4th row of data where col A > 3 and col B != 7
df[df[“A”] > 3 & df[“B”] != 7].iloc[3:5]

pandas will then return either a view or a copy of the dataframe. A view (shallow copy) references data from the original dataframe, while a copy (deep copy) is a separate instance of the same data.

It is difficult to predict which will be returned by the indexing operation, as it depends on the memory layout of the underlying array. How exactly the indexing is chained can lead to different __getitem__ and __setitem__ calls being issued under the hood. Reproducing an example below:

# Example borrowed from [1]
dfmi.loc[:, (‘one’, ‘second’)] = value
# becomes
dfmi.loc.__setitem__((slice(None), (‘one’, ‘second’)), value)

dfmi[‘one’][‘second’] = value
# becomes
dfmi.__getitem__(‘one’).__setitem__(‘second’, value)

Chain indexing inherently is not a problem, but assigning values using chained indexing, i.e. chained assignment, can be. Depending on the situation, chained assignment will either modify the original dataframe directly, or return a modified copy of the original dataframe. This can lead to insidious bugs when it is not obvious that chained indexing has occured.

Chained indexing can take places across a few lines of code:

# The following doesn’t look like chain indexing, does it?
dfa = df.loc[row1:row2, col1:col2]
…
…
dfa[row2] = dfa[row1].apply(fn)

If pandas did not raise a warning in this scenario, it would not be obvious that df is not modified by the second value assignment. This is why the SettingWithCopyWarning exists.

pandas docs [¹] go into this with more detail. The warning message helpfully links to it, which is great because if you search pandas settingwithcopywarning on Google, the docs page is easy to miss! At time of writing, it is the 7th result on the first page of Google, and is crowded out by blogposts and StackOverflow questions.

Peeking under the hood using the internal API

Chained indexing is a godsend of convenience for selecting the right data, but chained assignment is a minefield for assigning the correct values. The TowardsDataScience article in [²] has a nice example where inverting the order of chained indexing alone is the difference between whether an assignment to the original dataframe occurs or not:

# Example borrowed from [2]

# This updates `df`
df[“product_group”][df[“product_group”]==”PG4"] = “PG14”

# This doesn’t!
df[df[“product_group”]==”PG4"][“product_group”] = “PG14”

# pandas raises warnings for both
# the user needs to confirm the intended outcome

From this StackOverflow post, pd.DataFrame and pd.Series objects have _is_view and _is_copy attributes as part of their internal API. _is_view returns True if the object is a view, and False if the object is not. _is_copy stores either a weak reference to the dataframe it is copied from, or None if it is not associated to an existing dataframe.

Printing these internal attributes while poking around with chained assignment does reveal some interesting tidbits of info. On one hand, pandas uses _is_copy to decide if a SettingWithCopyWarning needs to be raised. On the other hand, modifying a dataframe with _is_view = True means that it will affect the original underlying dataframe.

Before we begin, a disclaimer: internal APIs are not meant to be accessed by the end user and are subject to change, use them at your own risk.

In [4]: pd.__version__
Out[4]: ‘1.3.3’

# Setting up convenience functions
In [5]: def make_clean_df():
     …:     df = pd.DataFrame({
     …:         “A”: [1, 2, 3, 4, 5],
     …:         “B”: [6, 7, 8, 9, 10],
     …:         “C”: [11, 12, 13, 14, 15],
     …:         }, index=range(5)
     …:     )
     …:     return df

In [6]: def show_attrs(obj):
     …:     print(f”view: {obj._is_view}, copy: {obj._is_copy}”)

We’ll start by showing the _is_view and _is_copy attributes of a few common indexing methods.

In [7]: df = make_clean_df()

In [8]: show_attrs(df.loc[3:5])
     …: show_attrs(df.iloc[3:5])
     …: show_attrs(df.loc[3:5, [“A”, “B”]])
     …: show_attrs(df.iloc[3:5, [0, 1]])
     …: show_attrs(df[“A”])
     …: show_attrs(df.loc[:, “A”])

view: True, copy: <weakref at 0x7f4d648b2590; to ‘DataFrame’ at 0x7f4d648b54c0>
view: True, copy: <weakref at 0x7f4d648b2590; to ‘DataFrame’ at 0x7f4d648b54c0>
view: False, copy: None
view: False, copy: <weakref at 0x7f4d648be770; dead>
view: True, copy: None
view: True, copy: None

Let’s break this down:

• Both df.loc[3:5] and df.iloc[3:5] returned views and have references to the original dataframe.
• For df.loc[3:5, [“A”, “B”]] and df.iloc[3:5, [0, 1]] , when the columns are additionally specified on top of the rows, copies of df are returned instead. Using .loc indexing has no references to the OG dataframe, while using iloc indexing results in a reference to a temporary dataframe that has been garbage collected, which is as good as None itself. We’ll see if this carries any significance.
• Referring to a column directly using either df[“A”] or df.loc[:, “A”] returns a view, with no reference to the original dataframe. It might have to do with the fact that each dataframe column is actually stored as a pd.Series.

What happens if we manually create copies of these indexed dataframes / series?

In [9]: show_attrs(df.loc[3:5].copy())
     …: show_attrs(df.iloc[3:5].copy())
     …: show_attrs(df.loc[3:5, [“A”, “B”]].copy())
     …: show_attrs(df.iloc[3:5, [0, 1]].copy())
     …: show_attrs(df[“A”].copy())
     …: show_attrs(df.loc[:, “A”].copy())

view: False, copy: None
view: False, copy: None
view: False, copy: None
view: False, copy: None
view: False, copy: None
view: False, copy: None

Explicitly calling .copy returns copies of data that have no reference to the original dataframe / series. Assigning data on these copies will not affect the original dataframe, and thus will not trigger SettingwithCopyWarnings. Given that df.loc[3:5, [“A”, “B”]] and df.iloc[3:5, [0, 1]] above have similar attributes, we can expect that their behaviour under chained assignment should be similar to explicitly created copies.

Next, we’ll try a few chained assignment scenarios.

Scenario 1: Specific rows indexed using loc

The following three chained assignments raise SettingWithCopyWarnings:

In [10]: df = make_clean_df()
      …: dfa = df.loc[3:5]
      …: show_attrs(dfa)

view: True, copy: <weakref at 0x7fba308565e0; to ‘DataFrame’ at 0x7fba3084eac0>

# (1a)
In [11]: dfa[dfa % 2 == 0] = 100

/tmp/ipykernel_34555/3321004726.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfa[dfa % 2 == 0] = 100
/home/tnwei/miniconda3/envs/ml/lib/python3.9/site-packages/pandas/core/
frame.py:3718: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 self._where(-key, value, inplace=True)

# (1b)
In [12]: dfa[“D”] = dfa[“B”] * 10

/tmp/ipykernel_34555/447367411.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfa[“D”] = dfa[“B”] * 10 # 1b

# (1c)
In [13]: dfa[“A”][3] = 10

/tmp/ipykernel_34555/1338713145.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfa[“A”][3] = 10

All of the value assignments took effect on dfa itself, but only (1a) and (1c) affected the original dataframe. (1b) did not.

In [14]: print(dfa)
 
  A  B  C   D
3 10 9  100 90
4 5  100 15 1000

In [15]: print(df)

  A  B  C
0 1  6  11
1 2  7  12
2 3  8  13
3 10 9  100
4 5  100 15

In addition, dfa is no longer a view, but a copy of the dataframe!

In [16]: show_attrs(dfa) # view changed to False

view: False, copy: <weakref at 0x7fba308565e0; to ‘DataFrame’ at 0x7fba3084eac0>

What this tells us is that pandas will convert a view to a copy when necessary. This further shows why figuring out chained assignment is inherently tricky, and is difficult to cater for automatically at the library level.

Scenario 2: Specific rows indexed using iloc

This is the same as scenario 1, but using iloc instead.

In [17]: df = make_clean_df()
      …: dfb = df.iloc[3:5]
      …: show_attrs(dfb)

view: True, copy: <weakref at 0x7fba30862040; to ‘DataFrame’ at 0x7fba30868c10>

# (1a)
In [18]: dfb[dfb % 2 == 0] = 100

/tmp/ipykernel_34555/734837801.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfb[dfb % 2 == 0] = 100
/home/tnwei/miniconda3/envs/ml/lib/python3.9/site-packages/pandas/core/
frame.py:3718: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 self._where(-key, value, inplace=True)

# (1b)
In [19]: dfb[“D”] = dfb[“B”] * 10

/tmp/ipykernel_34555/4288697762.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfb[“D”] = dfb[“B”] * 10

# (1c)
In [20]: dfb[“A”][3] = 10

/tmp/ipykernel_34555/2062795903.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/
stable/user_guide/indexing.html#returning-a-view-versus-a-copy
 dfb[“A”][3] = 10

The observed outcome is the same as Scenario 1.

In [21]: print(dfb)
  A  B  C   D
3 10 9  100 90
4 5  100 15 1000

In [22]: print(df)

  A  B   C
0 1  6   11
1 2  7   12
2 3  8   13
3 10 9   100
4 5  100 15

In [23]: show_attrs(dfb)

view: False, copy: <weakref at 0x7fba30862040; to ‘DataFrame’ at 0x7fba30868c10>

Scenario 3: Specific rows and columns indexed using loc

Same as Scenario 1, but the columns are specified as well.

In [24]: df = make_clean_df()
      …: dfc = df.loc[3:5, [“A”, “B”]]
      …: show_attrs(dfc)

view: False, copy: None

In [25]: dfc[dfc % 2 == 0] = 100 # No warnings raised
      …: dfc[“D”] = dfc[“B”] * 10
      …: dfc[“A”][3] = 10

No warnings raised. All changes took effect on `dfc` without impacting `df`.

In [26]: print(dfc)
 
  A  B  D
3 10 9  90
4 5  100 1000

In [27]: print(df)

  A  B   C
0 1  6   11
1 2  7   12
2 3  8   13
3 10 9   100
4 5  100 15

The chained assignment outcome is different, while the data indexed is the same as in Scenario 1. My guess is that a more complete description of the indexing operation prompted pandas to directly return a copy upfront, instead of a view that is linked to the original dataframe.

Scenario 4: Specific rows and columns indexed using iloc

This is similar to Scenario 3, but using iloc instead. Given the past few scenarios, it is no surprise that this scenario had the same outcome as Scenario 3.

In [28]: df = make_clean_df()
      …: dfd = df.iloc[3:5, [0, 1]]
      …: show_attrs(dfd)

view: False, copy: <weakref at 0x7fba306f29f0; dead>

In [29]: dfd[dfd % 2 == 0] = 100 # No warnings raised
      …: dfd[“D”] = dfd[“B”] * 10
      …: dfd[“A”][3] = 10

In [30]: print(dfd)

  A  B   D
3 10 9   90
4 5  100 1000

In [31]: print(df)

  A B  C
0 1 6  11
1 2 7  12
2 3 8  13
3 4 9  14
4 5 10 15

In addition, `dfd` discarded the reference to the garbage-collected dataframe at the end of this code.

In [32]: show_attrs(dfd)

view: False, copy: None

Scenario 5: Directly referring to a column of a dataframe

This scenario tests chained assignment on series.

In [33]: df = make_clean_df()
      …: dfe = df[“A”]
      …: show_attrs(dfe)

view: True, copy: None

In [34]: dfe[1] = 99999 # No warnings raised
      …: dfe.loc[2:4] = 88888

dfe remained a view of df[“A”]. All changes effected on dfe is reflected in df[“A”], which is still part of df. It appears that there’s not much to worry about for chained assignment on individual series.

In [35]: print(dfe)

0 1
1 99999
2 88888
3 88888
4 88888
Name: A, dtype: int64

In [36]: print(df)

  A     B C
0 1     6 11
1 99999 7 12
2 88888 8 13
3 88888 9 14
4 88888 10 15

In [37]: show_attrs(dfe)

view: True, copy: None

Getting rid of SettingWithCopyWarnings

SettingWithCopyWarnings pop up when pandas isn’t sure if you want value assignment to affect the original dataframe or not. Therefore, getting rid of these warnings entails avoiding ambiguity in value assignment. As seen from the code samples above, getting pandas to return copies with no reference to the original dataframe is a clean way to ensure that values will not be written to the original dataframe unintended.

I’ve found this to be a unifying thread across the solutions I came across when researching this topic. Summarizing them below:

Disabling the warnings

If you know what you’re doing and your code is behaving as intended, you can choose to suppress the warnings by disabling them [³]:

# Example borrowed from [³]

# Disables SettingWithCopyWarning globally
pd.set_option(‘mode.chained_assignment’, None)

# Resets the warning option to default
pd.reset_option(‘mode.chained_assignment’)

# Disables SettingWithCopyWarning locally within a context manager
with pd.option_context(‘mode.chained_assignment’, None):
    # YOUR CODE HERE

Alternatively, you can suppress the warnings by setting the dataframe _is_copy attribute to None [³].

# Example modified from [3]
In [38]: df = pd.DataFrame({
      …:     “A”: [1, 2, 3, 4, 5],
      …:     “B”: [6, 7, 8, 9, 10],
      …:     “C”: [11, 12, 13, 14, 15],
      …: }, index=range(5))
      …: 
      …: dfa = df.loc[3:5]
      …: print(dfa._is_copy)

<weakref at 0x7f4d64792810; to ‘DataFrame’ at 0x7f4d64784460>

In [39]: dfa._is_copy = None
      …: dfa[“D”] = dfa[“B”] * 10 # No warning raised

Remember that making the warnings go away doesn’t resolve wonky chained assignment issues. Chained assignment is a minefield where you might or might not step on a landmine. Disabling the warnings is like removing the minefield warning signs. Food for thought.

Making the warnings not show up in the first place

When you run into a SettingWithCopy warning, take a moment to trace the chained assignment and decide if you want to modify the original dataframe directly, or have values assigned to a copy of the dataframe.

Working on the original dataframe

Use .loc indexing to directly assign values to the dataframe.

# Modified from examples in [2]
In [40]: df = pd.DataFrame({
 …: “A”: [1, 2, 3, 4, 5],
 …: “B”: [6, 7, 8, 9, 10],
 …: “C”: [11, 12, 13, 14, 15],
 …: }, index=range(5))
 …:
 …: df.loc[df[“A”] % 2 != 0, “B”] = df.loc[df[“A”] % 2 != 0, “B”] + 0.5
 …: print(df)

A B C
0 1 6.5 11
1 2 7.0 12
2 3 8.5 13
3 4 9.0 14
4 5 10.5 15

pandas docs recommend this method for two reasons:

• using .loc is guaranteed to refer to the underlying dataframe it is called on. .iloc does not have this property.
• .loc indexing replaces what could be chain indexing into a single indexing step. If you refer to the example above under Chained assignment with views and copies, .loc indexing resolves chained indexing into a single __setitem__ call.

If you are selecting data using conditionals, you can consider returning a mask instead of a copy of the original dataframe. A mask is a boolean series or dataframe which can conveniently be used in .loc indexing, as the example below:

# Modified from examples in [5]
In [41]: df = pd.DataFrame({
      …:     “A”: [1, 2, 3, 4, 5],
      …:     “B”: [6, 7, 8, 9, 10],
      …:     “C”: [11, 12, 13, 14, 15],
      …: }, index=range(5))

In [42]: dfa = (df[“A”] <= 3) & (df[“C”] == 12)

In [43]: df.loc[dfa, “B”] = 99 # dfa can be fed into the loc index!

In [44]: print(df) # changes took place in the original dataframe

  A B  C
0 1 6  11
1 2 99 12
2 3 8  13
3 4 9  14
4 5 10 15

Working on the original dataframe directly can be tricky if existing indexing logic is complex. In that case, you can always use one of the methods from the next section to return a copy, then assign it back to the original dataframe [⁴].

Assigning values to an explicit copy of the dataframe

Use assign, where and replace:

In [45]: df = pd.DataFrame({
      …:     “A”: [1, 2, 3, 4, 5],
      …:     “B”: [6, 7, 8, 9, 10],
      …:     “C”: [11, 12, 13, 14, 15],
      …: }, index=range(5))

# 1. Use the `assign` method to add columns
In [46]: df = df.assign(D=df[“C”] * 10)
      …: df = df.assign(**{“D”: df[“C”] * 10}) # allows passing variables as names

# 2. Use the `where` method to select values using conditionals and replace them
# Modified from examples in [2]
In [47]: df[“B”] = df[“B”].where(
      …:     df[“A”] < 2, df[“B”] * 10
      …: )

# 3. Use the `replace` method to select and replace values in the dataframe
# Modified from examples in [2]
In [48]: df = df.replace({“A” : 1}, 100)

In [49]: print(df)
  A   B  C  D
0 100 6  11 110
1 2  70  12 120
2 3  80  13 130
3 4  90  14 140
4 5  100 15 150

Break down chained assignment steps into single assignments [⁵]:

# Examples borrowed from [4]
# Not these
df[“z”][mask] = 0
df.loc[mask][“z”] = 0

# But this
df.loc[mask, “z”] = 0

A less elegant but foolproof method is to manually create a copy of the original dataframe and work on it instead [²]. As long as you don’t introduce additional chained indexing, you will not see the SettingWithCopyWarning.

In [50]: df = pd.DataFrame({
      …:     “A”: [1, 2, 3, 4, 5],
      …:     “B”: [6, 7, 8, 9, 10],
      …:     “C”: [11, 12, 13, 14, 15],
      …: }, index=range(5))

In [51]: dfa = df.loc[3:5].copy() # Added .copy() here
      …: dfa.loc[3, “A”] = 10 # causes this line to raise no warning

Redoing some of the examples above without triggering SettingWithCopy warnings

Replace a chained assignment with where:

# (i)
df = make_clean_df()
dfa = df.loc[3:5]

# Original that raises warning
# dfa[dfa % 2 == 0] = 100

dfa = dfa.where(dfa % 2 != 0, 100) # df is not affected

Replace creating a new column on the indexed dataframe with assign:

# (ii) 
df = make_clean_df()

# Original that raises warning
# dfa[“D”] = dfa[“B”] * 10

dfa = dfa.assign(D=dfa[“B”]*10) # df is not affected

Creating a copy of the dataframe, before assigning values using .loc indexing:

# (iii)
df = make_clean_df()

# Original that raises warnings
# dfa = df.loc[3:5]
# dfa[“A”][3] = 10

# Create a copy then do loc indexing
dfa = df.loc[3:5].copy()
dfa.loc[3, “A”] = 10

Note that directly assigning values to dfa using .loc indexing will still raise a warning, as it is ambiguous if the assignment to dfa should also affect df.

Truly rooting out SettingWithCopyWarnings

Personally, I am a fan of promoting SettingWithCopyWarnings to SettingWithCopyExceptions for important scripts, using the following code:

pd.set_option(‘mode.chained_assignment’, “raise”)

Doing this forces chained assignment to be dealt with, rather than allowing warnings to accumulate.

In my experience, cleaning up notebooks with stderr clogged by SettingWithCopyWarnings is its special kind of zen. I wholeheartedly recommend it.

Original blogpost: SettingWithCopyWarnings in pandas

[¹]: Official `pandas` docs on chained assignment.
[https://pandas.pydata.org/docs/user_guide/indexing.html#returning-a-view-versus-a-copy](https://pandas.pydata.org/docs/user_guide/indexing.html#returning-a-view-versus-a-copy)

[²]: TowardsDataScience article that briefly touches on a few ways to deal with the SettingWithCopy warnings.
[https://scribe.rip/@towardsdatascience.com/3-solutions-for-the-setting-with-copy-warning-of-python-pandas-dfe15d62de08](https://scribe.rip/@towardsdatascience.com/3-solutions-for-the-setting-with-copy-warning-of-python-pandas-dfe15d62de08)

[³]: In-depth article on this topic by DataQuest. Notably, there is a section dedicated to the history of dealing with chained assignment in `pandas`.
[https://www.dataquest.io/blog/settingwithcopywarning/](https://www.dataquest.io/blog/settingwithcopywarning/)

[⁴]: StackOverflow post that contains more chained assignment examples. [https://stackoverflow.com/questions/48173980/pandas-knowing-when-an-operation-affects-the-original-dataframe](https://stackoverflow.com/questions/48173980/pandas-knowing-when-an-operation-affects-the-original-dataframe)

[⁵]: RealPython article covering this topic. For me, RealPython is a trusted goto reference second to official library docs. This article further goes into depth on the underlying view vs copy mechanisms in pandas, and in numpy, which pandas depends on.
[https://realpython.com/pandas-settingwithcopywarning/](https://realpython.com/pandas-settingwithcopywarning/)

SettingWithCopyWarning in pandas was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.