DEPR: passing mixed offsets with utc=False into to_datetime (#54014)

* add raising FutureWarning in _return_parsed_timezone_results and in array_to_datetime, fix tests

* correct the definition of _return_parsed_timezone_results, added a test for FutureWarning, fix tests

* fix tests in pandas/tests/extension/test_arrow.py

* correct the definition of _return_parsed_timezone_results

* fix an exanple in docs: Parsing a CSV with mixed timezones

* correct def _array_to_datetime_object, add a test for mixed format, fix errors in docs

* correct example in whatsnew/v0.24.0.rst and fix pylint failures

* correct str for message in FutureWarning in the test with format mixed

* fix an error in an example in whatsnew/v0.24.0.rst

* correct examples and the description of the param utc in docstring  of to_datetime, correct an example in whatsnew/v1.1.0.rst

* update whatsnew/v2.1.0.rst

* correct docstring for to_datetime, example in whatsnew/v0.24.0.rst, rename test functions

* refactor tests for to_datetime

* refactor test for to_datetime

* add example to whatsnew/v2.1.0.rst

* correct the example

* correct the example in whatsnew/v2.1.0.rst

* correct def _array_to_datetime_object and fix test for read_json

* add catch_warnings to filter the warning in test_read_datetime

* correct msg in catch_warnings to filter the warning in test_read_datetime

* catch the warning in test_from_csv_with_mixed_offsets

* reword whatsnew

* add catch_warnings to converter

* describe how to maintain the old behavior

* add an example how to get the old behavior and : correct the warning message

---------

Co-authored-by: MarcoGorelli <[email protected]>

Author: natmokval

doc/source/user_guide/io.rst

@@ -931,6 +931,8 @@ Parsing a CSV with mixed timezones
 pandas cannot natively represent a column or index with mixed timezones. If your CSV
 file contains columns with a mixture of timezones, the default result will be
 an object-dtype column with strings, even with ``parse_dates``.
+To parse the mixed-timezone values as a datetime column, read in as ``object`` dtype and
+then call :func:`to_datetime` with ``utc=True``.
 
 
 .. ipython:: python
@@ -939,14 +941,6 @@ an object-dtype column with strings, even with ``parse_dates``.
    a
    2000-01-01T00:00:00+05:00
    2000-01-01T00:00:00+06:00"""
-   df = pd.read_csv(StringIO(content), parse_dates=["a"])
-   df["a"]
-
-To parse the mixed-timezone values as a datetime column, read in as ``object`` dtype and
-then call :func:`to_datetime` with ``utc=True``.
-
-.. ipython:: python
-
    df = pd.read_csv(StringIO(content))
    df["a"] = pd.to_datetime(df["a"], utc=True)
    df["a"]

doc/source/whatsnew/v0.24.0.rst

@@ -632,13 +632,19 @@ Parsing datetime strings with the same UTC offset will preserve the UTC offset i
 Parsing datetime strings with different UTC offsets will now create an Index of
 ``datetime.datetime`` objects with different UTC offsets
 
-.. ipython:: python
+.. code-block:: ipython
+
+    In [59]: idx = pd.to_datetime(["2015-11-18 15:30:00+05:30",
+                                   "2015-11-18 16:30:00+06:30"])
+
+    In[60]: idx
+    Out[60]: Index([2015-11-18 15:30:00+05:30, 2015-11-18 16:30:00+06:30], dtype='object')
+
+    In[61]: idx[0]
+    Out[61]: Timestamp('2015-11-18 15:30:00+0530', tz='UTC+05:30')
 
-    idx = pd.to_datetime(["2015-11-18 15:30:00+05:30",
-                          "2015-11-18 16:30:00+06:30"])
-    idx
-    idx[0]
-    idx[1]
+    In[62]: idx[1]
+    Out[62]: Timestamp('2015-11-18 16:30:00+0630', tz='UTC+06:30')
 
 Passing ``utc=True`` will mimic the previous behavior but will correctly indicate
 that the dates have been converted to UTC
@@ -673,15 +679,22 @@ Parsing mixed-timezones with :func:`read_csv`
 
 *New behavior*
 
-.. ipython:: python
+.. code-block:: ipython
+
+   In[64]: import io
+
+   In[65]: content = """\
+      ...: a
+      ...: 2000-01-01T00:00:00+05:00
+      ...: 2000-01-01T00:00:00+06:00"""
+
+   In[66]: df = pd.read_csv(io.StringIO(content), parse_dates=['a'])
 
-   import io
-   content = """\
-   a
-   2000-01-01T00:00:00+05:00
-   2000-01-01T00:00:00+06:00"""
-   df = pd.read_csv(io.StringIO(content), parse_dates=['a'])
-   df.a
+   In[67]: df.a
+   Out[67]:
+   0   2000-01-01 00:00:00+05:00
+   1   2000-01-01 00:00:00+06:00
+   Name: a, Length: 2, dtype: object
 
 As can be seen, the ``dtype`` is object; each value in the column is a string.
 To convert the strings to an array of datetimes, the ``date_parser`` argument

doc/source/whatsnew/v1.1.0.rst

@@ -208,7 +208,14 @@ For example:
     tz_strs = ["2010-01-01 12:00:00 +0100", "2010-01-01 12:00:00 -0100",
                "2010-01-01 12:00:00 +0300", "2010-01-01 12:00:00 +0400"]
     pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z', utc=True)
-    pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z')
+
+.. code-block:: ipython
+
+   In[37]: pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z')
+   Out[37]:
+   Index([2010-01-01 12:00:00+01:00, 2010-01-01 12:00:00-01:00,
+          2010-01-01 12:00:00+03:00, 2010-01-01 12:00:00+04:00],
+         dtype='object')
 
 .. _whatsnew_110.grouper_resample_origin:
 

doc/source/whatsnew/v2.1.0.rst

@@ -295,8 +295,53 @@ Other API changes
 .. ---------------------------------------------------------------------------
 .. _whatsnew_210.deprecations:
 
-Deprecations
-~~~~~~~~~~~~
+Deprecate parsing datetimes with mixed time zones
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Parsing datetimes with mixed time zones is deprecated and shows a warning unless user passes ``utc=True`` to :func:`to_datetime` (:issue:`50887`)
+
+*Previous behavior*:
+
+.. code-block:: ipython
+
+  In [7]: data = ["2020-01-01 00:00:00+06:00", "2020-01-01 00:00:00+01:00"]
+
+  In [8]:  pd.to_datetime(data, utc=False)
+  Out[8]:
+  Index([2020-01-01 00:00:00+06:00, 2020-01-01 00:00:00+01:00], dtype='object')
+
+*New behavior*:
+
+.. code-block:: ipython
+
+  In [9]: pd.to_datetime(data, utc=False)
+  FutureWarning:
+    In a future version of pandas, parsing datetimes with mixed time zones will raise
+    a warning unless `utc=True`. Please specify `utc=True` to opt in to the new behaviour
+    and silence this warning. To create a `Series` with mixed offsets and `object` dtype,
+    please use `apply` and `datetime.datetime.strptime`.
+  Index([2020-01-01 00:00:00+06:00, 2020-01-01 00:00:00+01:00], dtype='object')
+
+In order to silence this warning and avoid an error in a future version of pandas,
+please specify ``utc=True``:
+
+.. ipython:: python
+
+    data = ["2020-01-01 00:00:00+06:00", "2020-01-01 00:00:00+01:00"]
+    pd.to_datetime(data, utc=True)
+
+To create a ``Series`` with mixed offsets and ``object`` dtype, please use ``apply``
+and ``datetime.datetime.strptime``:
+
+.. ipython:: python
+
+    import datetime as dt
+
+    data = ["2020-01-01 00:00:00+06:00", "2020-01-01 00:00:00+01:00"]
+    pd.Series(data).apply(lambda x: dt.datetime.strptime(x, '%Y-%m-%d %H:%M:%S%z'))
+
+Other Deprecations
+~~~~~~~~~~~~~~~~~~
 - Deprecated 'broadcast_axis' keyword in :meth:`Series.align` and :meth:`DataFrame.align`, upcast before calling ``align`` with ``left = DataFrame({col: left for col in right.columns}, index=right.index)`` (:issue:`51856`)
 - Deprecated 'downcast' keyword in :meth:`Index.fillna` (:issue:`53956`)
 - Deprecated 'fill_method' and 'limit' keywords in :meth:`DataFrame.pct_change`, :meth:`Series.pct_change`, :meth:`DataFrameGroupBy.pct_change`, and :meth:`SeriesGroupBy.pct_change`, explicitly call ``ffill`` or ``bfill`` before calling ``pct_change`` instead (:issue:`53491`)

pandas/_libs/tslib.pyx

@@ -620,6 +620,7 @@ cdef _array_to_datetime_object(
     # 1) NaT or NaT-like values
     # 2) datetime strings, which we return as datetime.datetime
     # 3) special strings - "now" & "today"
+    unique_timezones = set()
     for i in range(n):
         # Analogous to: val = values[i]
         val = <object>(<PyObject**>cnp.PyArray_MultiIter_DATA(mi, 1))[0]
@@ -649,6 +650,7 @@ cdef _array_to_datetime_object(
                     tzinfo=tsobj.tzinfo,
                     fold=tsobj.fold,
                 )
+                unique_timezones.add(tsobj.tzinfo)
 
             except (ValueError, OverflowError) as ex:
                 ex.args = (f"{ex}, at position {i}", )
@@ -666,6 +668,16 @@ cdef _array_to_datetime_object(
 
         cnp.PyArray_MultiIter_NEXT(mi)
 
+    if len(unique_timezones) > 1:
+        warnings.warn(
+            "In a future version of pandas, parsing datetimes with mixed time "
+            "zones will raise a warning unless `utc=True`. "
+            "Please specify `utc=True` to opt in to the new behaviour "
+            "and silence this warning. To create a `Series` with mixed offsets and "
+            "`object` dtype, please use `apply` and `datetime.datetime.strptime`",
+            FutureWarning,
+            stacklevel=find_stack_level(),
+        )
     return oresult_nd, None
 
 

pandas/core/tools/datetimes.py

@@ -340,6 +340,7 @@ def _return_parsed_timezone_results(
     tz_result : Index-like of parsed dates with timezone
     """
     tz_results = np.empty(len(result), dtype=object)
+    non_na_timezones = set()
     for zone in unique(timezones):
         mask = timezones == zone
         dta = DatetimeArray(result[mask]).tz_localize(zone)
@@ -348,8 +349,20 @@ def _return_parsed_timezone_results(
                 dta = dta.tz_localize("utc")
             else:
                 dta = dta.tz_convert("utc")
+        else:
+            if not dta.isna().all():
+                non_na_timezones.add(zone)
         tz_results[mask] = dta
-
+    if len(non_na_timezones) > 1:
+        warnings.warn(
+            "In a future version of pandas, parsing datetimes with mixed time "
+            "zones will raise a warning unless `utc=True`. Please specify `utc=True` "
+            "to opt in to the new behaviour and silence this warning. "
+            "To create a `Series` with mixed offsets and `object` dtype, "
+            "please use `apply` and `datetime.datetime.strptime`",
+            FutureWarning,
+            stacklevel=find_stack_level(),
+        )
     return Index(tz_results, name=name)
 
 
@@ -772,6 +785,14 @@ def to_datetime(
           offsets (typically, daylight savings), see :ref:`Examples
           <to_datetime_tz_examples>` section for details.
 
+        .. warning::
+
+            In a future version of pandas, parsing datetimes with mixed time
+            zones will raise a warning unless `utc=True`.
+            Please specify `utc=True` to opt in to the new behaviour
+            and silence this warning. To create a `Series` with mixed offsets and
+            `object` dtype, please use `apply` and `datetime.datetime.strptime`.
+
         See also: pandas general documentation about `timezone conversion and
         localization
         <https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html
@@ -993,19 +1014,33 @@ def to_datetime(
 
     - However, timezone-aware inputs *with mixed time offsets* (for example
       issued from a timezone with daylight savings, such as Europe/Paris)
-      are **not successfully converted** to a :class:`DatetimeIndex`. Instead a
-      simple :class:`Index` containing :class:`datetime.datetime` objects is
-      returned:
-
-    >>> pd.to_datetime(['2020-10-25 02:00 +0200', '2020-10-25 04:00 +0100'])
+      are **not successfully converted** to a :class:`DatetimeIndex`.
+      Parsing datetimes with mixed time zones will show a warning unless
+      `utc=True`. If you specify `utc=False` the warning below will be shown
+      and a simple :class:`Index` containing :class:`datetime.datetime`
+      objects will be returned:
+
+    >>> pd.to_datetime(['2020-10-25 02:00 +0200',
+    ...                 '2020-10-25 04:00 +0100'])  # doctest: +SKIP
+    FutureWarning: In a future version of pandas, parsing datetimes with mixed
+    time zones will raise a warning unless `utc=True`. Please specify `utc=True`
+    to opt in to the new behaviour and silence this warning. To create a `Series`
+    with mixed offsets and `object` dtype, please use `apply` and
+    `datetime.datetime.strptime`.
     Index([2020-10-25 02:00:00+02:00, 2020-10-25 04:00:00+01:00],
           dtype='object')
 
     - A mix of timezone-aware and timezone-naive inputs is also converted to
       a simple :class:`Index` containing :class:`datetime.datetime` objects:
 
     >>> from datetime import datetime
-    >>> pd.to_datetime(["2020-01-01 01:00:00-01:00", datetime(2020, 1, 1, 3, 0)])
+    >>> pd.to_datetime(["2020-01-01 01:00:00-01:00",
+    ...                 datetime(2020, 1, 1, 3, 0)])  # doctest: +SKIP
+    FutureWarning: In a future version of pandas, parsing datetimes with mixed
+    time zones will raise a warning unless `utc=True`. Please specify `utc=True`
+    to opt in to the new behaviour and silence this warning. To create a `Series`
+    with mixed offsets and `object` dtype, please use `apply` and
+    `datetime.datetime.strptime`.
     Index([2020-01-01 01:00:00-01:00, 2020-01-01 03:00:00], dtype='object')
 
     |

pandas/io/json/_json.py

@@ -1312,7 +1312,14 @@ def _try_convert_to_date(self, data):
         date_units = (self.date_unit,) if self.date_unit else self._STAMP_UNITS
         for date_unit in date_units:
             try:
-                new_data = to_datetime(new_data, errors="raise", unit=date_unit)
+                with warnings.catch_warnings():
+                    warnings.filterwarnings(
+                        "ignore",
+                        ".*parsing datetimes with mixed time "
+                        "zones will raise a warning",
+                        category=FutureWarning,
+                    )
+                    new_data = to_datetime(new_data, errors="raise", unit=date_unit)
             except (ValueError, OverflowError, TypeError):
                 continue
             return new_data, True

pandas/io/parsers/base_parser.py

@@ -1144,37 +1144,59 @@ def converter(*date_cols, col: Hashable):
                 date_format.get(col) if isinstance(date_format, dict) else date_format
             )
 
-            result = tools.to_datetime(
-                ensure_object(strs),
-                format=date_fmt,
-                utc=False,
-                dayfirst=dayfirst,
-                errors="ignore",
-                cache=cache_dates,
-            )
+            with warnings.catch_warnings():
+                warnings.filterwarnings(
+                    "ignore",
+                    ".*parsing datetimes with mixed time zones will raise a warning",
+                    category=FutureWarning,
+                )
+                result = tools.to_datetime(
+                    ensure_object(strs),
+                    format=date_fmt,
+                    utc=False,
+                    dayfirst=dayfirst,
+                    errors="ignore",
+                    cache=cache_dates,
+                )
             if isinstance(result, DatetimeIndex):
                 arr = result.to_numpy()
                 arr.flags.writeable = True
                 return arr
             return result._values
         else:
             try:
-                result = tools.to_datetime(
-                    date_parser(*(unpack_if_single_element(arg) for arg in date_cols)),
-                    errors="ignore",
-                    cache=cache_dates,
-                )
+                with warnings.catch_warnings():
+                    warnings.filterwarnings(
+                        "ignore",
+                        ".*parsing datetimes with mixed time zones "
+                        "will raise a warning",
+                        category=FutureWarning,
+                    )
+                    result = tools.to_datetime(
+                        date_parser(
+                            *(unpack_if_single_element(arg) for arg in date_cols)
+                        ),
+                        errors="ignore",
+                        cache=cache_dates,
+                    )
                 if isinstance(result, datetime.datetime):
                     raise Exception("scalar parser")
                 return result
             except Exception:
-                return tools.to_datetime(
-                    parsing.try_parse_dates(
-                        parsing.concat_date_cols(date_cols),
-                        parser=date_parser,
-                    ),
-                    errors="ignore",
-                )
+                with warnings.catch_warnings():
+                    warnings.filterwarnings(
+                        "ignore",
+                        ".*parsing datetimes with mixed time zones "
+                        "will raise a warning",
+                        category=FutureWarning,
+                    )
+                    return tools.to_datetime(
+                        parsing.try_parse_dates(
+                            parsing.concat_date_cols(date_cols),
+                            parser=date_parser,
+                        ),
+                        errors="ignore",
+                    )
 
     return converter
 

pandas/tests/indexes/datetimes/test_constructors.py

@@ -300,8 +300,10 @@ def test_construction_index_with_mixed_timezones(self):
         assert not isinstance(result, DatetimeIndex)
 
         msg = "DatetimeIndex has mixed timezones"
+        msg_depr = "parsing datetimes with mixed time zones will raise a warning"
         with pytest.raises(TypeError, match=msg):
-            DatetimeIndex(["2013-11-02 22:00-05:00", "2013-11-03 22:00-06:00"])
+            with tm.assert_produces_warning(FutureWarning, match=msg_depr):
+                DatetimeIndex(["2013-11-02 22:00-05:00", "2013-11-03 22:00-06:00"])
 
         # length = 1
         result = Index([Timestamp("2011-01-01")], name="idx")