Troubles with date_range and DST

[barsanges]

Code Sample

>>> pd.date_range('2018-10-23', '2018-11-06', freq='7D', tz='Europe/Paris')
DatetimeIndex(['2018-10-23 00:00:00+02:00', '2018-10-29 23:00:00+01:00', '2018-11-05 23:00:00+01:00'], dtype='datetime64[ns, Europe/Paris]', freq='7D')

>>> pd.date_range('2018-03-25', '2018-04-09', freq='7D', tz='Europe/Paris')
DatetimeIndex(['2018-03-25 00:00:00+01:00', '2018-04-01 01:00:00+02:00', '2018-04-08 01:00:00+02:00'], dtype='datetime64[ns, Europe/Paris]', freq='7D')

Problem description

It seems that the D frequency is somewhat broken around DST dates. The code above gives the same result as pd.date_range(..., freq='168H', tz='Europe/Paris'), but it should be different.

Expected Output

>>> pd.date_range('2018-10-23', '2018-11-06', freq='7D', tz='Europe/Paris')
DatetimeIndex(['2018-10-23 00:00:00+02:00', '2018-10-30 00:00:00+01:00', '2018-11-06 00:00:00+01:00'], dtype='datetime64[ns, Europe/Paris]', freq='7D')

>>> pd.date_range('2018-03-25', '2018-04-09', freq='7D', tz='Europe/Paris')
DatetimeIndex(['2018-03-25 00:00:00+01:00', '2018-04-01 00:00:00+02:00', '2018-04-08 00:00:00+02:00'], dtype='datetime64[ns, Europe/Paris]', freq='7D')

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 3.16.0-5-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: fr_FR.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.3
setuptools: 39.0.1
Cython: None
numpy: 1.14.2
scipy: 1.0.1
pyarrow: None
xarray: None
IPython: 6.3.0
sphinx: None
patsy: None
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[mroeschke]

Yup, This looks incorrect since the date_range docstring states:

freq : string or DateOffset, default ‘D’ (calendar daily)

# Same as freq='7D'
In [10]: pd.date_range('2018-10-23', '2018-11-06', freq=7 * pd.tseries.offsets.Day(), tz='Europe/Paris')
Out[10]:
DatetimeIndex(['2018-10-23 00:00:00+02:00', '2018-10-29 23:00:00+01:00',
               '2018-11-05 23:00:00+01:00'],
              dtype='datetime64[ns, Europe/Paris]', freq='7D')

# Respects calendar day
In [11]: pd.date_range('2018-10-23', '2018-11-06', freq=pd.DateOffset(days=7), tz='Europe/Paris')
Out[11]:
DatetimeIndex(['2018-10-23 00:00:00+02:00', '2018-10-30 00:00:00+01:00',
               '2018-11-06 00:00:00+01:00'],
              dtype='datetime64[ns, Europe/Paris]', freq='<DateOffset: days=7>')

[jreback]

this is a duplicate issue from quite a while ago if u can do a search

[mroeschke]

xref #16980

[jreback]

closing as master issue #20633

[kalikgod]

Related to this issue, I have found doing a pandas.date_range over the repeated hour at the end of DST produces duplicate hours.

start = pd.Timestamp('2019-11-03 00:00:00', tz='US/Central')

end = pd.Timestamp('2019-11-04 00:00:00', tz='US/Central')

pd.date_range(start=start, end=end, freq='H')

DatetimeIndex(['2019-11-03 00:00:00-05:00', '2019-11-03 01:00:00-06:00',
'2019-11-03 01:00:00-06:00', '2019-11-03 02:00:00-06:00',
'2019-11-03 03:00:00-06:00', '2019-11-03 04:00:00-06:00',
'2019-11-03 05:00:00-06:00', '2019-11-03 06:00:00-06:00',
'2019-11-03 07:00:00-06:00', '2019-11-03 08:00:00-06:00',
'2019-11-03 09:00:00-06:00', '2019-11-03 10:00:00-06:00',
'2019-11-03 11:00:00-06:00', '2019-11-03 12:00:00-06:00',
'2019-11-03 13:00:00-06:00', '2019-11-03 14:00:00-06:00',
'2019-11-03 15:00:00-06:00', '2019-11-03 16:00:00-06:00',
'2019-11-03 17:00:00-06:00', '2019-11-03 18:00:00-06:00',
'2019-11-03 19:00:00-06:00', '2019-11-03 20:00:00-06:00',
'2019-11-03 21:00:00-06:00', '2019-11-03 22:00:00-06:00',
'2019-11-03 23:00:00-06:00', '2019-11-04 00:00:00-06:00'],
dtype='datetime64[ns, US/Central]', freq='H')

Is there a work around to get reliable timestamp for every hour in a day?

[kalikgod]

After making a manual work around, I found that pd.Dataframe.index resets the time offset back to the incorrect value.

df
Out[26]:
Empty DataFrame
Columns: []
Index: [2019-11-03 00:00:00-05:00, 2019-11-03 01:00:00-05:00, 2019-11-03 01:00:00-06:00, 2019-11-03 02:00:00-06:00, 2019-11-03 03:00:00-06:00, 2019-11-03 04:00:00-06:00, 2019-11-03 05:00:00-06:00, 2019-11-03 06:00:00-06:00, 2019-11-03 07:00:00-06:00, 2019-11-03 08:00:00-06:00, 2019-11-03 09:00:00-06:00, 2019-11-03 10:00:00-06:00, 2019-11-03 11:00:00-06:00, 2019-11-03 12:00:00-06:00, 2019-11-03 13:00:00-06:00, 2019-11-03 14:00:00-06:00, 2019-11-03 15:00:00-06:00, 2019-11-03 16:00:00-06:00, 2019-11-03 17:00:00-06:00, 2019-11-03 18:00:00-06:00, 2019-11-03 19:00:00-06:00, 2019-11-03 20:00:00-06:00, 2019-11-03 21:00:00-06:00, 2019-11-03 22:00:00-06:00, 2019-11-03 23:00:00-06:00, 2019-11-04 00:00:00-06:00]

df.index
Out[27]:
DatetimeIndex(['2019-11-03 00:00:00-05:00', '2019-11-03 01:00:00-06:00',
'2019-11-03 01:00:00-06:00', '2019-11-03 02:00:00-06:00',
'2019-11-03 03:00:00-06:00', '2019-11-03 04:00:00-06:00',
'2019-11-03 05:00:00-06:00', '2019-11-03 06:00:00-06:00',
'2019-11-03 07:00:00-06:00', '2019-11-03 08:00:00-06:00',
'2019-11-03 09:00:00-06:00', '2019-11-03 10:00:00-06:00',
'2019-11-03 11:00:00-06:00', '2019-11-03 12:00:00-06:00',
'2019-11-03 13:00:00-06:00', '2019-11-03 14:00:00-06:00',
'2019-11-03 15:00:00-06:00', '2019-11-03 16:00:00-06:00',
'2019-11-03 17:00:00-06:00', '2019-11-03 18:00:00-06:00',
'2019-11-03 19:00:00-06:00', '2019-11-03 20:00:00-06:00',
'2019-11-03 21:00:00-06:00', '2019-11-03 22:00:00-06:00',
'2019-11-03 23:00:00-06:00', '2019-11-04 00:00:00-06:00'],
dtype='datetime64[ns, US/Central]', freq=None)