Skip to content

pandas_datareader failing to get all options for '^ndx' from yahoo #190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vishalapte opened this issue Mar 14, 2016 · 2 comments
Closed

pandas_datareader failing to get all options for '^ndx' from yahoo #190

vishalapte opened this issue Mar 14, 2016 · 2 comments

Comments

@vishalapte
Copy link

The code below works for over 500 symbols but doesn't work for ^ndx.

In [1]: from pandas_datareader import data as pddata

In [2]: foo = pddata.Options('^ndx', 'yahoo')

In [3]: foo.expiry_dates
Out[3]: 
[datetime.date(2016, 3, 18),
 datetime.date(2016, 3, 24),
 datetime.date(2016, 4, 1),
 datetime.date(2016, 4, 8),
 datetime.date(2016, 4, 15),
 datetime.date(2016, 4, 22),
 datetime.date(2016, 4, 29),
 datetime.date(2016, 5, 20),
 datetime.date(2016, 6, 17),
 datetime.date(2016, 9, 16),
 datetime.date(2016, 12, 16),
 datetime.date(2017, 6, 16),
 datetime.date(2017, 12, 15),
 datetime.date(2018, 12, 21)]

In [4]: for d in foo.expiry_dates:
    print(d)
    foo.get_options_data(expiry=d)
   ...:     
2016-03-18
2016-03-24
2016-04-01
2016-04-08
2016-04-15
2016-04-22
2016-04-29
2016-05-20
2016-06-17
2016-09-16
2016-12-16

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-5-dad98abfb543> in <module>()
      1 for d in foo.expiry_dates:
      2     print d
----> 3     foo.get_options_data(expiry=d)
      4 

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in get_options_data(self, month, year, expiry)
    146         return concat([f(month, year, expiry)
    147                        for f in (self.get_put_data,
--> 148                                  self.get_call_data)]).sortlevel()
    149 
    150     def _get_option_frames_from_yahoo(self, expiry):

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in get_put_data(self, month, year, expiry)
    360         """
    361         expiry = self._try_parse_dates(year, month, expiry)
--> 362         return self._get_data_in_date_range(expiry, put=True, call=False)
    363 
    364     def get_near_stock_price(self, above_below=2, call=True, put=False,

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _get_data_in_date_range(self, dates, call, put)
    606                     frame = getattr(self, nam)
    607                 except AttributeError:
--> 608                     frame = self._get_option_data(expiry=expiry_date, name=name)
    609                 data.append(frame)
    610 

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _get_option_data(self, expiry, name)
    229             frames = getattr(self, frame_name)
    230         except AttributeError:
--> 231             frames = self._get_option_frames_from_yahoo(expiry)
    232 
    233         option_data = frames[name]

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _get_option_frames_from_yahoo(self, expiry)
    150     def _get_option_frames_from_yahoo(self, expiry):
    151         url = self._yahoo_url_from_expiry(expiry)
--> 152         option_frames = self._option_frames_from_url(url)
    153         frame_name = '_frames' + self._expiry_to_string(expiry)
    154         setattr(self, frame_name, option_frames)

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _option_frames_from_url(self, url)
    173 
    174         root = self._parse_url(url)
--> 175         calls = root.xpath('//*[@id="optionsCallsTable"]/div[2]/div/table')[0]
    176         puts = root.xpath('//*[@id="optionsPutsTable"]/div[2]/div/table')[0]
    177 

IndexError: list index out of range

In [9]: import pandas_datareader

In [10]: pandas_datareader.version
Out[10]: '0.2.1'
@femtotrader
Copy link
Contributor

Here is my traceback (which is slightly different than yours probably as I'm using Python 3)

In [15]: for d in foo.expiry_dates:
        print(d)
        foo.get_options_data(expiry=d)
   ....:
2016-03-24
2016-04-01
2016-04-08
2016-04-15
2016-04-22
2016-04-29
2016-05-20
2016-06-17
2016-09-16
2016-12-16
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_data_in_date_range(self, dates, call, put)
    610                 try:  # Try to access on the instance
--> 611                     frame = getattr(self, nam)
    612                 except AttributeError:

AttributeError: 'Options' object has no attribute 'puts161216'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_option_data(self, expiry, name)
    233         try:
--> 234             frames = getattr(self, frame_name)
    235         except AttributeError:

AttributeError: 'Options' object has no attribute '_frames161216'

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _option_frames_from_url(self, url)
    177         try:
--> 178             calls = root.xpath('//*[@id="optionsCallsTable"]/div[2]/div/table')[0]
    179             puts = root.xpath('//*[@id="optionsPutsTable"]/div[2]/div/table')[0]

IndexError: list index out of range

During handling of the above exception, another exception occurred:

RemoteDataError                           Traceback (most recent call last)
<ipython-input-15-42e2169d1e3b> in <module>()
      1 for d in foo.expiry_dates:
      2         print(d)
----> 3         foo.get_options_data(expiry=d)
      4

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in get_options_data(self, month, year, expiry)
    148         return concat([f(month, year, expiry)
    149                        for f in (self.get_put_data,
--> 150                                  self.get_call_data)]).sortlevel()
    151
    152     def _get_option_frames_from_yahoo(self, expiry):

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in <listcomp>(.0)
    147         """
    148         return concat([f(month, year, expiry)
--> 149                        for f in (self.get_put_data,
    150                                  self.get_call_data)]).sortlevel()
    151

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in get_put_data(self, month, year, expiry)
    365         """
    366         expiry = self._try_parse_dates(year, month, expiry)
--> 367         return self._get_data_in_date_range(expiry, put=True, call=False)
    368
    369     def get_near_stock_price(self, above_below=2, call=True, put=False,

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_data_in_date_range(self, dates, call, put)
    611                     frame = getattr(self, nam)
    612                 except AttributeError:
--> 613                     frame = self._get_option_data(expiry=expiry_date, name=name)
    614                 data.append(frame)
    615

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_option_data(self, expiry, name)
    234             frames = getattr(self, frame_name)
    235         except AttributeError:
--> 236             frames = self._get_option_frames_from_yahoo(expiry)
    237
    238         option_data = frames[name]

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_option_frames_from_yahoo(self, expiry)
    152     def _get_option_frames_from_yahoo(self, expiry):
    153         url = self._yahoo_url_from_expiry(expiry)
--> 154         option_frames = self._option_frames_from_url(url)
    155         frame_name = '_frames' + self._expiry_to_string(expiry)
    156         setattr(self, frame_name, option_frames)

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _option_frames_from_url(self, url)
    179             puts = root.xpath('//*[@id="optionsPutsTable"]/div[2]/div/table')[0]
    180         except IndexError:
--> 181             raise RemoteDataError('Option Table not available for url: %s' % url)
    182
    183         if not hasattr(self, 'underlying_price'):

RemoteDataError: Option Table not available for url: http://finance.yahoo.com/q/op?s=^NDX&date=1481846400

Such traceback can occur by doing

foo.get_options_data(expiry=datetime.date(2016, 12, 16))

but I' m getting

In [22]: foo.get_options_data(expiry=datetime.date(2017, 12, 15))
Out[22]:
                                             Last     Bid     Ask  Chg  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000  2294.9  2369.0  2384.5    0
                  put  NDX171215P01980000    28.1    20.0    23.5    0
2000   2015-12-17 call NDX171215C02000000  2343.8  2350.0  2365.5    0
                  put  NDX171215P02000000     0.0    26.0    32.0    0
2100   2015-12-17 call NDX171215C02100000  2182.5  2256.1  2272.5    0
...                                           ...     ...     ...  ...
6950   2015-12-17 put  NDX171215P06950000     0.0  2629.4  2650.0    0
6975   2015-12-17 call NDX171215C06975000     0.0     0.0     5.0    0
                  put  NDX171215P06975000  2260.9  2543.7  2563.0    0
7000   2015-12-17 call NDX171215C07000000     0.0     0.0     5.0    0
                  put  NDX171215P07000000     0.0  2677.8  2699.0    0

                                           PctChg  Vol  Open_Int      IV Root  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000       0    6        10  0.0000  NDX
                  put  NDX171215P01980000       0    0         1  0.3526  NDX
2000   2015-12-17 call NDX171215C02000000       0    1         9  0.0000  NDX
                  put  NDX171215P02000000       0    0         0  0.3710  NDX
2100   2015-12-17 call NDX171215C02100000       0    1        29  0.0000  NDX
...                                           ...  ...       ...     ...  ...
6950   2015-12-17 put  NDX171215P06950000       0    0         0  0.2837  NDX
6975   2015-12-17 call NDX171215C06975000       0    0         0  0.1551  NDX
                  put  NDX171215P06975000       0    0         1  0.0994  NDX
7000   2015-12-17 call NDX171215C07000000       0    0         0  0.1561  NDX
                  put  NDX171215P07000000       0    0         0  0.2859  NDX

                                          IsNonstandard Underlying  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000          True       ^NDX
                  put  NDX171215P01980000          True       ^NDX
2000   2015-12-17 call NDX171215C02000000          True       ^NDX
                  put  NDX171215P02000000          True       ^NDX
2100   2015-12-17 call NDX171215C02100000          True       ^NDX
...                                                 ...        ...
6950   2015-12-17 put  NDX171215P06950000          True       ^NDX
6975   2015-12-17 call NDX171215C06975000          True       ^NDX
                  put  NDX171215P06975000          True       ^NDX
7000   2015-12-17 call NDX171215C07000000          True       ^NDX
                  put  NDX171215P07000000          True       ^NDX

                                           Underlying_Price  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000           4410.83
                  put  NDX171215P01980000           4410.83
2000   2015-12-17 call NDX171215C02000000           4410.83
                  put  NDX171215P02000000           4410.83
2100   2015-12-17 call NDX171215C02100000           4410.83
...                                                     ...
6950   2015-12-17 put  NDX171215P06950000           4410.83
6975   2015-12-17 call NDX171215C06975000           4410.83
                  put  NDX171215P06975000           4410.83
7000   2015-12-17 call NDX171215C07000000           4410.83
                  put  NDX171215P07000000           4410.83

                                                   Quote_Time
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000 2016-03-19 17:15:00
                  put  NDX171215P01980000 2016-03-19 17:15:00
2000   2015-12-17 call NDX171215C02000000 2016-03-19 17:15:00
                  put  NDX171215P02000000 2016-03-19 17:15:00
2100   2015-12-17 call NDX171215C02100000 2016-03-19 17:15:00
...                                                       ...
6950   2015-12-17 put  NDX171215P06950000 2016-03-19 17:15:00
6975   2015-12-17 call NDX171215C06975000 2016-03-19 17:15:00
                  put  NDX171215P06975000 2016-03-19 17:15:00
7000   2015-12-17 call NDX171215C07000000 2016-03-19 17:15:00
                  put  NDX171215P07000000 2016-03-19 17:15:00

[758 rows x 13 columns]

I notice that for 2016/12/16
http://finance.yahoo.com/q/op?s=^NDX&date=1481846400
doesn't display any data

but for 2017/12/15
http://finance.yahoo.com/q/op?s=^NDX&date=1513296000
I'm getting data

maybe in such a case (of empty page) we might return an empty DataFrame like

pd.DataFrame(columns=['Last', 'Bid', 'Ask', 'Chg', 'PctChg', 'Vol', 'Open_Int', 'IV', 'Root',
   ....:        'IsNonstandard', 'Underlying', 'Underlying_Price', 'Quote_Time'])

an other approach (on your side) might be to use a try/except like:

from pandas_datareader.yahoo.options import RemoteDataError

for d in foo.expiry_dates:
    print(d)
    try:
        foo.get_options_data(expiry=d)
    except RemoteDataError:
        print("Can't get options data for %s" % d)

@davidastephens
Copy link
Member

This looks to work now after #244. Please re-open if still broken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vishalapte @femtotrader @davidastephens