Skip to content

DEPR: 'epoch' date format in to_json #57987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Apr 19, 2024
Merged

DEPR: 'epoch' date format in to_json #57987

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
339b2e0
deprecate epoch date format in to_json
Aloqeely Mar 24, 2024
41156fd
Fix tests
Aloqeely Mar 25, 2024
8fee0ea
Remove examples using epoch
Aloqeely Mar 25, 2024
80cd7fe
Deprecate in v3.0.0
Aloqeely Mar 25, 2024
5300d7e
Warn for using default format
Aloqeely Apr 6, 2024
f59aa80
Fix detection logic
Aloqeely Apr 6, 2024
9938eb5
Keep default and expect warning
Aloqeely Apr 9, 2024
75df145
Add deprecation test
Aloqeely Apr 9, 2024
f9c39b4
Final changes
Aloqeely Apr 13, 2024
36d3cf4
Code review suggestions
Aloqeely Apr 16, 2024
50d6d99
Stop use of filterwarnings
Aloqeely Apr 16, 2024
8e4b95c
Make pre-commit happy
Aloqeely Apr 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 2 additions & 9 deletions doc/source/user_guide/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1949,13 +1949,6 @@ Writing in ISO date format, with microseconds:
json = dfd.to_json(date_format="iso", date_unit="us")
json

Epoch timestamps, in seconds:

.. ipython:: python

json = dfd.to_json(date_format="epoch", date_unit="s")
json

Writing to a file, with a date index and a date column:

.. ipython:: python
Expand All @@ -1965,7 +1958,7 @@ Writing to a file, with a date index and a date column:
dfj2["ints"] = list(range(5))
dfj2["bools"] = True
dfj2.index = pd.date_range("20130101", periods=5)
dfj2.to_json("test.json")
dfj2.to_json("test.json", date_format="iso")

with open("test.json") as fh:
print(fh.read())
Expand Down Expand Up @@ -2140,7 +2133,7 @@ Dates written in nanoseconds need to be read back in nanoseconds:
.. ipython:: python

from io import StringIO
json = dfj2.to_json(date_unit="ns")
json = dfj2.to_json(date_format="iso", date_unit="ns")

# Try to parse timestamps as milliseconds -> Won't Work
dfju = pd.read_json(StringIO(json), date_unit="ms")
Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ Other Deprecations
- Deprecated :meth:`Timestamp.utcnow`, use ``Timestamp.now("UTC")`` instead (:issue:`56680`)
- Deprecated allowing non-keyword arguments in :meth:`Series.to_markdown` except ``buf``. (:issue:`57280`)
- Deprecated allowing non-keyword arguments in :meth:`Series.to_string` except ``buf``. (:issue:`57280`)
- Deprecated using ``epoch`` date format in :meth:`DataFrame.to_json` and :meth:`Series.to_json`, use ``iso`` instead. (:issue:`57063`)
-

.. ---------------------------------------------------------------------------
Expand Down
21 changes: 21 additions & 0 deletions pandas/core/generic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2328,6 +2328,11 @@ def to_json(
'iso' = ISO8601. The default depends on the `orient`. For
``orient='table'``, the default is 'iso'. For all other orients,
the default is 'epoch'.

.. deprecated:: 3.0.0
'epoch' date format is deprecated and will be removed in a future
version, please use 'iso' instead.

double_precision : int, default 10
The number of decimal places to use when encoding
floating point values. The possible maximal value is 15.
Expand Down Expand Up @@ -2530,6 +2535,22 @@ def to_json(
date_format = "iso"
elif date_format is None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think need to warn for anything that is not currently iso, including when date_format is None (although the message will be different)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add a warning for the date_format=None case that previously defaulted to "epoch"; this should in the future default to "iso"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the to_json use cases don't involve dates and wouldn't be affected by the date_format value, throwing a warning in these cases might be unnecessary, essentially they will need to pass date_format='iso' for no reason to silence this warning, are you sure we should do this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to be more specific we need to warn when date_format=None and we actually serialize timestamp types. I agree no point in warning if a DataFrame has no timestamp type, but if users are relying on the default epoch behavior they need to be warned of the change

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WillAyd curious how would users get the old behavior? It would be good to add that in the warning message

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old behavior as in just an integer? I think the problem with that is it was an implementation detail of pandas spilling out into the JSON serializer. Historically our timestamps were exclusively nanoseconds since the Unix epoch, but with all the work @jbrockmendel has been doing that is no longer true (and _usually not true).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old behavior as in just an integer?

Yeah. Just checking if we can still offer a suggestion for a migration path if they want to keep the old behavior

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. Especially with our auto-inferencing of resolutions I don't see how it would be usable at all roundtripping through JSON

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK sounds good

date_format = "epoch"
dtypes = self.dtypes if self.ndim == 2 else [self.dtype]
if any(lib.is_np_dtype(dtype, "mM") for dtype in dtypes):
warnings.warn(
"The default 'epoch' date format is deprecated and will be removed "
"in a future version, please use 'iso' date format instead.",
FutureWarning,
stacklevel=find_stack_level(),
)
elif date_format == "epoch":
# GH#57063
warnings.warn(
"'epoch' date format is deprecated and will be removed in a future "
"version, please use 'iso' date format instead.",
FutureWarning,
stacklevel=find_stack_level(),
)

config.is_nonnegative_int(indent)
indent = indent or 0
Expand Down
11 changes: 8 additions & 3 deletions pandas/tests/io/json/test_json_table_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,12 +451,17 @@ def test_to_json_categorical_index(self):
assert result == expected

def test_date_format_raises(self, df_table):
msg = (
error_msg = (
"Trying to write with `orient='table'` and `date_format='epoch'`. Table "
"Schema requires dates to be formatted with `date_format='iso'`"
)
with pytest.raises(ValueError, match=msg):
df_table.to_json(orient="table", date_format="epoch")
warning_msg = (
"'epoch' date format is deprecated and will be removed in a future "
"version, please use 'iso' date format instead."
)
with pytest.raises(ValueError, match=error_msg):
with tm.assert_produces_warning(FutureWarning, match=warning_msg):
df_table.to_json(orient="table", date_format="epoch")

# others work
df_table.to_json(orient="table", date_format="iso")
Expand Down
Loading