Working with pandas

One of the most important features of xarray is the ability to convert to and from pandas objects to interact with the rest of the PyData ecosystem. For example, for plotting labeled data, we highly recommend using the visualization built in to pandas itself or provided by the pandas aware libraries such as Seaborn.

Hierarchical and tidy data

Tabular data is easiest to work with when it meets the criteria for tidy data:

• Each column holds a different variable.

• Each rows holds a different observation.

In this “tidy data” format, we can represent any Dataset and DataArray in terms of DataFrame and Series, respectively (and vice-versa). The representation works by flattening non-coordinates to 1D, and turning the tensor product of coordinate indexes into a pandas.MultiIndex.

Dataset and DataFrame

To convert any dataset to a DataFrame in tidy form, use the Dataset.to_dataframe() method:

ds = xr.Dataset(
    {"foo": (("x", "y"), np.random.randn(2, 3))},
    coords={
        "x": [10, 20],
        "y": ["a", "b", "c"],
        "along_x": ("x", np.random.randn(2)),
        "scalar": 123,
    },
)
ds

<xarray.Dataset> Size: 100B
Dimensions:  (x: 2, y: 3)
Coordinates:
  * x        (x) int64 16B 10 20
  * y        (y) <U1 12B 'a' 'b' 'c'
    along_x  (x) float64 16B 0.1192 -1.044
    scalar   int64 8B 123
Data variables:
    foo      (x, y) float64 48B 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732

xarray.Dataset

• Dimensions:
  • x: 2
  • y: 3
• Coordinates: (4)
  • x
    (x)
    int64
    10 20

    array([10, 20])

  • y
    (y)
    <U1
    'a' 'b' 'c'

    array(['a', 'b', 'c'], dtype='<U1')

  • along_x
    (x)
    float64
    0.1192 -1.044

    array([ 0.11920871, -1.04423597])

  • scalar
    ()
    int64
    123

    array(123)

• Data variables: (1)
  • foo
    (x, y)
    float64
    0.4691 -0.2829 ... 1.212 -0.1732

    array([[ 0.4691123 , -0.28286334, -1.5090585 ],
           [-1.13563237,  1.21211203, -0.17321465]])

• Indexes: (2)
  • x
    PandasIndex

    PandasIndex(Index([10, 20], dtype='int64', name='x'))

  • y
    PandasIndex

    PandasIndex(Index(['a', 'b', 'c'], dtype='object', name='y'))

• Attributes: (0)

df = ds.to_dataframe()
df

		foo	along_x	scalar
x	y
10	a	0.469112	0.119209	123
	b	-0.282863	0.119209	123
	c	-1.509059	0.119209	123
20	a	-1.135632	-1.044236	123
	b	1.212112	-1.044236	123
	c	-0.173215	-1.044236	123

We see that each variable and coordinate in the Dataset is now a column in the DataFrame, with the exception of indexes which are in the index. To convert the DataFrame to any other convenient representation, use DataFrame methods like reset_index(), stack() and unstack().

For datasets containing dask arrays where the data should be lazily loaded, see the Dataset.to_dask_dataframe() method.

To create a Dataset from a DataFrame, use the Dataset.from_dataframe() class method or the equivalent pandas.DataFrame.to_xarray() method:

xr.Dataset.from_dataframe(df)

<xarray.Dataset> Size: 184B
Dimensions:  (x: 2, y: 3)
Coordinates:
  * x        (x) int64 16B 10 20
  * y        (y) object 24B 'a' 'b' 'c'
Data variables:
    foo      (x, y) float64 48B 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
    along_x  (x, y) float64 48B 0.1192 0.1192 0.1192 -1.044 -1.044 -1.044
    scalar   (x, y) int64 48B 123 123 123 123 123 123

xarray.Dataset

• Dimensions:
  • x: 2
  • y: 3
• Coordinates: (2)
  • x
    (x)
    int64
    10 20

    array([10, 20])

  • y
    (y)
    object
    'a' 'b' 'c'

    array(['a', 'b', 'c'], dtype=object)

• Data variables: (3)
  • foo
    (x, y)
    float64
    0.4691 -0.2829 ... 1.212 -0.1732

    array([[ 0.4691123 , -0.28286334, -1.5090585 ],
           [-1.13563237,  1.21211203, -0.17321465]])

  • along_x
    (x, y)
    float64
    0.1192 0.1192 ... -1.044 -1.044

    array([[ 0.11920871,  0.11920871,  0.11920871],
           [-1.04423597, -1.04423597, -1.04423597]])

  • scalar
    (x, y)
    int64
    123 123 123 123 123 123

    array([[123, 123, 123],
           [123, 123, 123]])

• Indexes: (2)
  • x
    PandasIndex

    PandasIndex(Index([10, 20], dtype='int64', name='x'))

  • y
    PandasIndex

    PandasIndex(Index(['a', 'b', 'c'], dtype='object', name='y'))

• Attributes: (0)

Notice that the dimensions of variables in the Dataset have now expanded after the round-trip conversion to a DataFrame. This is because every object in a DataFrame must have the same indices, so we need to broadcast the data of each array to the full size of the new MultiIndex.

Likewise, all the coordinates (other than indexes) ended up as variables, because pandas does not distinguish non-index coordinates.

DataArray and Series

DataArray objects have a complementary representation in terms of a Series. Using a Series preserves the Dataset to DataArray relationship, because DataFrames are dict-like containers of Series. The methods are very similar to those for working with DataFrames:

s = ds["foo"].to_series()
s

x   y
10  a    0.469112
    b   -0.282863
    c   -1.509059
20  a   -1.135632
    b    1.212112
    c   -0.173215
Name: foo, dtype: float64

# or equivalently, with Series.to_xarray()
xr.DataArray.from_series(s)

<xarray.DataArray 'foo' (x: 2, y: 3)> Size: 48B
array([[ 0.4691123 , -0.28286334, -1.5090585 ],
       [-1.13563237,  1.21211203, -0.17321465]])
Coordinates:
  * x        (x) int64 16B 10 20
  * y        (y) object 24B 'a' 'b' 'c'

xarray.DataArray

'foo'

• x: 2
• y: 3

• 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732

  array([[ 0.4691123 , -0.28286334, -1.5090585 ],
         [-1.13563237,  1.21211203, -0.17321465]])

• Coordinates: (2)
  • x
    (x)
    int64
    10 20

    array([10, 20])

  • y
    (y)
    object
    'a' 'b' 'c'

    array(['a', 'b', 'c'], dtype=object)

• Indexes: (2)
  • x
    PandasIndex

    PandasIndex(Index([10, 20], dtype='int64', name='x'))

  • y
    PandasIndex

    PandasIndex(Index(['a', 'b', 'c'], dtype='object', name='y'))

• Attributes: (0)

Both the from_series and from_dataframe methods use reindexing, so they work even if the hierarchical index is not a full tensor product:

s[::2]

x   y
10  a    0.469112
    c   -1.509059
20  b    1.212112
Name: foo, dtype: float64

s[::2].to_xarray()

<xarray.DataArray 'foo' (x: 2, y: 3)> Size: 48B
array([[ 0.4691123 ,         nan, -1.5090585 ],
       [        nan,  1.21211203,         nan]])
Coordinates:
  * x        (x) int64 16B 10 20
  * y        (y) object 24B 'a' 'b' 'c'

xarray.DataArray

'foo'

• x: 2
• y: 3

• 0.4691 nan -1.509 nan 1.212 nan

  array([[ 0.4691123 ,         nan, -1.5090585 ],
         [        nan,  1.21211203,         nan]])

• Coordinates: (2)
  • x
    (x)
    int64
    10 20

    array([10, 20])

  • y
    (y)
    object
    'a' 'b' 'c'

    array(['a', 'b', 'c'], dtype=object)

• Indexes: (2)
  • x
    PandasIndex

    PandasIndex(Index([10, 20], dtype='int64', name='x'))

  • y
    PandasIndex

    PandasIndex(Index(['a', 'b', 'c'], dtype='object', name='y'))

• Attributes: (0)

Lossless and reversible conversion

The previous Dataset example shows that the conversion is not reversible (lossy roundtrip) and that the size of the Dataset increases.

Particularly after a roundtrip, the following deviations are noted:

• a non-dimension Dataset coordinate is converted into variable

• a non-dimension DataArray coordinate is not converted

• dtype is not always the same (e.g. “str” is converted to “object”)

• attrs metadata is not conserved

To avoid these problems, the third-party ntv-pandas library offers lossless and reversible conversions between Dataset/ DataArray and pandas DataFrame objects.

This solution is particularly interesting for converting any DataFrame into a Dataset (the converter finds the multidimensional structure hidden by the tabular structure).

The ntv-pandas examples show how to improve the conversion for the previous Dataset example and for more complex examples.

Multi-dimensional data

Tidy data is great, but it sometimes you want to preserve dimensions instead of automatically stacking them into a MultiIndex.

DataArray.to_pandas() is a shortcut that lets you convert a DataArray directly into a pandas object with the same dimensionality, if available in pandas (i.e., a 1D array is converted to a Series and 2D to DataFrame):

arr = xr.DataArray(
    np.random.randn(2, 3), coords=[("x", [10, 20]), ("y", ["a", "b", "c"])]
)
df = arr.to_pandas()
df

y	a	b	c
x
10	-0.861849	-2.104569	-0.494929
20	1.071804	0.721555	-0.706771

To perform the inverse operation of converting any pandas objects into a data array with the same shape, simply use the DataArray constructor:

xr.DataArray(df)

<xarray.DataArray (x: 2, y: 3)> Size: 48B
array([[-0.86184896, -2.10456922, -0.49492927],
       [ 1.07180381,  0.72155516, -0.70677113]])
Coordinates:
  * x        (x) int64 16B 10 20
  * y        (y) object 24B 'a' 'b' 'c'

xarray.DataArray

• x: 2
• y: 3

• -0.8618 -2.105 -0.4949 1.072 0.7216 -0.7068

  array([[-0.86184896, -2.10456922, -0.49492927],
         [ 1.07180381,  0.72155516, -0.70677113]])

• Coordinates: (2)
  • x
    (x)
    int64
    10 20

    array([10, 20])

  • y
    (y)
    object
    'a' 'b' 'c'

    array(['a', 'b', 'c'], dtype=object)

• Indexes: (2)
  • x
    PandasIndex

    PandasIndex(Index([10, 20], dtype='int64', name='x'))

  • y
    PandasIndex

    PandasIndex(Index(['a', 'b', 'c'], dtype='object', name='y'))

• Attributes: (0)

Both the DataArray and Dataset constructors directly convert pandas objects into xarray objects with the same shape. This means that they preserve all use of multi-indexes:

index = pd.MultiIndex.from_arrays(
    [["a", "a", "b"], [0, 1, 2]], names=["one", "two"]
)
df = pd.DataFrame({"x": 1, "y": 2}, index=index)
ds = xr.Dataset(df)
ds

<xarray.Dataset> Size: 120B
Dimensions:  (dim_0: 3)
Coordinates:
  * dim_0    (dim_0) object 24B MultiIndex
  * one      (dim_0) object 24B 'a' 'a' 'b'
  * two      (dim_0) int64 24B 0 1 2
Data variables:
    x        (dim_0) int64 24B 1 1 1
    y        (dim_0) int64 24B 2 2 2

xarray.Dataset

• Dimensions:
  • dim_0: 3
• Coordinates: (3)
  • dim_0
    (dim_0)
    object
    MultiIndex

    [3 values with dtype=object]

  • one
    (dim_0)
    object
    'a' 'a' 'b'

    [3 values with dtype=object]

  • two
    (dim_0)
    int64
    0 1 2

    [3 values with dtype=int64]

• Data variables: (2)
  • x
    (dim_0)
    int64
    1 1 1

    array([1, 1, 1])

  • y
    (dim_0)
    int64
    2 2 2

    array([2, 2, 2])

• Indexes: (1)
  • dim_0
    one
    two
    PandasMultiIndex

    PandasIndex(MultiIndex([('a', 0),
                ('a', 1),
                ('b', 2)],
               name='dim_0'))

• Attributes: (0)

However, you will need to set dimension names explicitly, either with the dims argument on in the DataArray constructor or by calling rename on the new object.

Transitioning from pandas.Panel to xarray

Panel, pandas’ data structure for 3D arrays, was always a second class data structure compared to the Series and DataFrame. To allow pandas developers to focus more on its core functionality built around the DataFrame, pandas removed Panel in favor of directing users who use multi-dimensional arrays to xarray.

Xarray has most of Panel’s features, a more explicit API (particularly around indexing), and the ability to scale to >3 dimensions with the same interface.

As discussed in the data structures section of the docs, there are two primary data structures in xarray: DataArray and Dataset. You can imagine a DataArray as a n-dimensional pandas Series (i.e. a single typed array), and a Dataset as the DataFrame equivalent (i.e. a dict of aligned DataArray objects).

So you can represent a Panel, in two ways:

• As a 3-dimensional DataArray,

• Or as a Dataset containing a number of 2-dimensional DataArray objects.

Let’s take a look:

data = np.random.default_rng(0).random((2, 3, 4))
items = list("ab")
major_axis = list("mno")
minor_axis = pd.date_range(start="2000", periods=4, name="date")

With old versions of pandas (prior to 0.25), this could stored in a Panel:

pd.Panel(data, items, major_axis, minor_axis)

<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: a to b
Major_axis axis: m to o
Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-04 00:00:00

To put this data in a DataArray, write:

array = xr.DataArray(data, [items, major_axis, minor_axis])
array

<xarray.DataArray (dim_0: 2, dim_1: 3, date: 4)> Size: 192B
array([[[0.63696169, 0.26978671, 0.04097352, 0.01652764],
        [0.81327024, 0.91275558, 0.60663578, 0.72949656],
        [0.54362499, 0.93507242, 0.81585355, 0.0027385 ]],

       [[0.85740428, 0.03358558, 0.72965545, 0.17565562],
        [0.86317892, 0.54146122, 0.29971189, 0.42268722],
        [0.02831967, 0.12428328, 0.67062441, 0.64718951]]])
Coordinates:
  * dim_0    (dim_0) <U1 8B 'a' 'b'
  * dim_1    (dim_1) <U1 12B 'm' 'n' 'o'
  * date     (date) datetime64[ns] 32B 2000-01-01 2000-01-02 ... 2000-01-04

xarray.DataArray

• dim_0: 2
• dim_1: 3
• date: 4

• 0.637 0.2698 0.04097 0.01653 0.8133 ... 0.02832 0.1243 0.6706 0.6472

  array([[[0.63696169, 0.26978671, 0.04097352, 0.01652764],
          [0.81327024, 0.91275558, 0.60663578, 0.72949656],
          [0.54362499, 0.93507242, 0.81585355, 0.0027385 ]],
  
         [[0.85740428, 0.03358558, 0.72965545, 0.17565562],
          [0.86317892, 0.54146122, 0.29971189, 0.42268722],
          [0.02831967, 0.12428328, 0.67062441, 0.64718951]]])

• Coordinates: (3)
  • dim_0
    (dim_0)
    <U1
    'a' 'b'

    array(['a', 'b'], dtype='<U1')

  • dim_1
    (dim_1)
    <U1
    'm' 'n' 'o'

    array(['m', 'n', 'o'], dtype='<U1')

  • date
    (date)
    datetime64[ns]
    2000-01-01 ... 2000-01-04

    array(['2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000',
           '2000-01-03T00:00:00.000000000', '2000-01-04T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (3)
  • dim_0
    PandasIndex

    PandasIndex(Index(['a', 'b'], dtype='object', name='dim_0'))

  • dim_1
    PandasIndex

    PandasIndex(Index(['m', 'n', 'o'], dtype='object', name='dim_1'))

  • date
    PandasIndex

    PandasIndex(DatetimeIndex(['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04'], dtype='datetime64[ns]', name='date', freq='D'))

• Attributes: (0)

As you can see, there are three dimensions (each is also a coordinate). Two of the axes of were unnamed, so have been assigned dim_0 and dim_1 respectively, while the third retains its name date.

You can also easily convert this data into Dataset:

array.to_dataset(dim="dim_0")

<xarray.Dataset> Size: 236B
Dimensions:  (dim_1: 3, date: 4)
Coordinates:
  * dim_1    (dim_1) <U1 12B 'm' 'n' 'o'
  * date     (date) datetime64[ns] 32B 2000-01-01 2000-01-02 ... 2000-01-04
Data variables:
    a        (dim_1, date) float64 96B 0.637 0.2698 0.04097 ... 0.8159 0.002739
    b        (dim_1, date) float64 96B 0.8574 0.03359 0.7297 ... 0.6706 0.6472

xarray.Dataset

• Dimensions:
  • dim_1: 3
  • date: 4
• Coordinates: (2)
  • dim_1
    (dim_1)
    <U1
    'm' 'n' 'o'

    array(['m', 'n', 'o'], dtype='<U1')

  • date
    (date)
    datetime64[ns]
    2000-01-01 ... 2000-01-04

    array(['2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000',
           '2000-01-03T00:00:00.000000000', '2000-01-04T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (2)
  • a
    (dim_1, date)
    float64
    0.637 0.2698 ... 0.8159 0.002739

    array([[0.63696169, 0.26978671, 0.04097352, 0.01652764],
           [0.81327024, 0.91275558, 0.60663578, 0.72949656],
           [0.54362499, 0.93507242, 0.81585355, 0.0027385 ]])

  • b
    (dim_1, date)
    float64
    0.8574 0.03359 ... 0.6706 0.6472

    array([[0.85740428, 0.03358558, 0.72965545, 0.17565562],
           [0.86317892, 0.54146122, 0.29971189, 0.42268722],
           [0.02831967, 0.12428328, 0.67062441, 0.64718951]])

• Indexes: (2)
  • dim_1
    PandasIndex

    PandasIndex(Index(['m', 'n', 'o'], dtype='object', name='dim_1'))

  • date
    PandasIndex

    PandasIndex(DatetimeIndex(['2000-01-01', '2000-01-02', '2000-01-03', '2000-01-04'], dtype='datetime64[ns]', name='date', freq='D'))

• Attributes: (0)

Here, there are two data variables, each representing a DataFrame on panel’s items axis, and labeled as such. Each variable is a 2D array of the respective values along the items dimension.

While the xarray docs are relatively complete, a few items stand out for Panel users:

• A DataArray’s data is stored as a numpy array, and so can only contain a single type. As a result, a Panel that contains DataFrame objects with multiple types will be converted to dtype=object. A Dataset of multiple DataArray objects each with its own dtype will allow original types to be preserved.

• Indexing is similar to pandas, but more explicit and leverages xarray’s naming of dimensions.

• Because of those features, making much higher dimensional data is very practical.

• Variables in Dataset objects can use a subset of its dimensions. For example, you can have one dataset with Person x Score x Time, and another with Person x Score.

• You can use coordinates are used for both dimensions and for variables which _label_ the data variables, so you could have a coordinate Age, that labelled the Person dimension of a Dataset of Person x Score x Time.

While xarray may take some getting used to, it’s worth it! If anything is unclear, please post an issue on GitHub or StackOverflow, and we’ll endeavor to respond to the specific case or improve the general docs.