BUG: Fix float formatting when a string is passed as float_format arg #22308

tomneep · 2018-08-13T10:05:39Z

If a string float_format argument is passed by the user to to_string, to_latex, to_html, etc, then disable fixed width, so that now e.g.

df.to_string(float_format='%.3f')

will give the same result as

df.to_string(float_format=lambda x: '%.3f' % x)

closes to_latex with float_format doesn't work like expected #21625 and to_html w/ float_format="%.0f" renders 100.0 as 1 #22270
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Fixes GH21625 and GH22270

jreback

would need a whatsnew entry as well

jreback · 2018-08-13T11:34:08Z

pandas/tests/io/formats/test_format.py

@@ -1328,6 +1328,12 @@ def test_to_string_float_formatting(self):
                        '1  2.512000e-01')
        assert df_s == expected

+        df = DataFrame({'x': [0.19999, 100.0]})


can you make a new test, pls add the issue number as a comment

jreback · 2018-08-13T11:35:31Z

pandas/io/formats/format.py

@@ -974,6 +974,7 @@ def __init__(self, *args, **kwargs):
        # float_format is expected to be a string
        # formatter should be used to pass a function
        if self.float_format is not None and self.formatter is None:
+            self.fixed_width = False


can you see if u can change fixed_width to a cached property instead of setting it (need to remove from the signature as well)

is this possible?

There is a good chance I'm not understanding the comment, but I'm not sure we can set this as a cached property (using the cache_readonly decorator?) since it is set in the base class. As I say I might be missing something...

you can set it in each of the subclasses. I just don't think we actually need this. try doing this as a property first to see if it works.

jreback · 2018-08-13T11:37:00Z

can you add the test from #21625 as well

codecov · 2018-08-13T11:44:23Z

Codecov Report

Merging #22308 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #22308      +/-   ##
==========================================
+ Coverage   92.29%   92.29%   +<.01%     
==========================================
  Files         161      161              
  Lines       51497    51501       +4     
==========================================
+ Hits        47530    47534       +4     
  Misses       3967     3967

Flag	Coverage Δ
#multiple	`90.69% <100%> (ø)`	⬆️
#single	`42.42% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/formats/format.py	`97.76% <100%> (ø)`	⬆️
pandas/io/pytables.py	`92.3% <0%> (-0.05%)`	⬇️
pandas/core/reshape/pivot.py	`96.55% <0%> (ø)`	⬆️
pandas/io/json/normalize.py	`96.87% <0%> (ø)`	⬆️
pandas/io/packers.py	`88.08% <0%> (ø)`	⬆️
pandas/core/util/hashing.py	`98.4% <0%> (ø)`	⬆️
pandas/core/algorithms.py	`95.11% <0%> (ø)`	⬆️
pandas/util/_decorators.py	`91.34% <0%> (ø)`	⬆️
pandas/tseries/offsets.py	`96.98% <0%> (ø)`	⬆️
pandas/core/indexes/datetimes.py	`96.2% <0%> (ø)`	⬆️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5c90e5...a9666bd. Read the comment docs.

jreback · 2018-08-14T00:07:13Z

doc/source/whatsnew/v0.24.0.txt

-
+- :func:`DataFrame.to_string()`, :func:`DataFrame.to_html()` and
+  :func:`DataFrame.to_latex()` will correctly format output when a string is
+  passed as the ``float_format`` argument (:issue:`21625` and :issue:`22270`)


use a ',' rather than 'and'

jreback · 2018-08-14T00:07:35Z

pandas/io/formats/format.py

@@ -974,6 +974,7 @@ def __init__(self, *args, **kwargs):
        # float_format is expected to be a string
        # formatter should be used to pass a function
        if self.float_format is not None and self.formatter is None:
+            self.fixed_width = False


is this possible?

jreback · 2018-09-15T12:26:57Z

can you rebase

jreback · 2018-09-18T12:50:03Z

can you rebase and see if you can address comments above

tomneep · 2018-09-18T13:10:54Z

Sorry I am away at the moment. I can have a look at this next week.

jreback · 2018-11-23T03:23:52Z

can you merge master and update to comments

pep8speaks · 2018-11-23T14:14:55Z

Hello @tomneep! Thanks for updating the PR.

There are no PEP8 issues in the file pandas/io/formats/format.py !
There are no PEP8 issues in the file pandas/tests/io/formats/test_format.py !
There are no PEP8 issues in the file pandas/tests/io/formats/test_to_html.py !
There are no PEP8 issues in the file pandas/tests/io/formats/test_to_latex.py !

Comment last updated on November 26, 2018 at 12:27 Hours UTC

jreback · 2018-11-23T14:57:33Z

pandas/tests/io/formats/test_format.py

@@ -1359,6 +1359,18 @@ def test_to_string_float_formatting(self):
                        '1  2.512000e-01')
        assert df_s == expected

+    def test_to_string_float_format_no_fixed_width(self):


you would need to add tests for to_latex and to_html in the appropriate files

tomneep · 2018-11-23T15:02:43Z

Changes above are just merging with the current master.
Sorry it's taken me so long to get back to this, I have some time now.

With fresh eyes I agree this probably isn't the best way to solve this but I'm a bit stuck how better to proceed.

grepping the code I see that fixed_width isn't used outside of the FloatArrayFormatter (other than appearing in the base class __init__), but there are a couple of instances of FloatArrayFormatter created in pandas/core/indexes/numeric.py and pandas/core/internals/blocks.py with fixed_width=False.

A slightly different way of doing essentially the same thing (but without touching the *ArrayFormatters) would be to change format_array (https://github.com/pandas-dev/pandas/blob/master/pandas/io/formats/format.py#L846). For example, adding fixed_width=float_format is None to the fmt_klass instance has the same effect, with all tests passing. This is maybe slightly nicer, as you could still directly create a FloatArrayFormatter with a value of float_format and fixed_width=True.

Any suggestions would be welcome.

jreback · 2018-11-23T21:12:56Z

the way the formatters is done now is not super flexible, so your change is ok. if you can add tests for latex & html would be good.

WillAyd · 2018-11-26T12:40:51Z

pandas/tests/io/formats/data/gh21625_expected_output.html

@@ -0,0 +1,14 @@
+<table border="1" class="dataframe">


Don't have a very strong opinion atm but wondering if we want dedicated files for tests this small. Could maybe include HTML as part of the test or alternately just test for the existence of the appropriate format in the result or the expression.

This doesn't need to be addressed in this PR but just bringing up as a general discussion point

WillAyd

lgtm

jreback · 2018-11-26T13:59:26Z

thanks!

…pandas-dev#22308)

BUG: Fix float formatting when a string is passed as float_format arg

bebe152

Fixes GH21625 and GH22270

jreback requested changes Aug 13, 2018

View reviewed changes

jreback added the Output-Formatting __repr__ of pandas objects, to_string label Aug 13, 2018

tomneep added 2 commits August 13, 2018 14:14

Put new float_format test cases into separate test

bac5ee5

Add whatsnew entry

e554d07

jreback requested changes Aug 14, 2018

View reviewed changes

Comma to and in whatsnew entry

2069f3d

tomneep added 2 commits November 23, 2018 10:39

Rebase

a7ddf80

Modify whatsnew entry

5fa9ff8

jreback requested changes Nov 23, 2018

View reviewed changes

tomneep added 2 commits November 24, 2018 12:25

Adding tests to test_to_html and test_to_latex

bda9961

Add new line to end of test_to_html.py

a9666bd

WillAyd reviewed Nov 26, 2018

View reviewed changes

WillAyd approved these changes Nov 26, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Nov 26, 2018

jreback approved these changes Nov 26, 2018

View reviewed changes

jreback merged commit b7294dd into pandas-dev:master Nov 26, 2018

tomneep deleted the fl_format_fix branch November 26, 2018 14:21

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG: Fix float formatting when a string is passed as float_format arg (…

3da91a1

…pandas-dev#22308)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG: Fix float formatting when a string is passed as float_format arg (…

f459a48

…pandas-dev#22308)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Fix float formatting when a string is passed as float_format arg #22308

BUG: Fix float formatting when a string is passed as float_format arg #22308

tomneep commented Aug 13, 2018 •

edited by jreback

Loading

jreback left a comment

jreback Aug 13, 2018

jreback Aug 13, 2018

jreback Aug 14, 2018

tomneep Aug 14, 2018

jreback Aug 14, 2018

jreback commented Aug 13, 2018

codecov bot commented Aug 13, 2018 •

edited

Loading

jreback Aug 14, 2018

jreback Aug 14, 2018

jreback commented Sep 15, 2018

jreback commented Sep 18, 2018

tomneep commented Sep 18, 2018

jreback commented Nov 23, 2018

pep8speaks commented Nov 23, 2018 •

edited

Loading

jreback Nov 23, 2018

tomneep commented Nov 23, 2018

jreback commented Nov 23, 2018

WillAyd Nov 26, 2018

WillAyd left a comment

jreback commented Nov 26, 2018

BUG: Fix float formatting when a string is passed as float_format arg #22308

BUG: Fix float formatting when a string is passed as float_format arg #22308

Conversation

tomneep commented Aug 13, 2018 • edited by jreback Loading

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 13, 2018

codecov bot commented Aug 13, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Sep 15, 2018

jreback commented Sep 18, 2018

tomneep commented Sep 18, 2018

jreback commented Nov 23, 2018

pep8speaks commented Nov 23, 2018 • edited Loading

Comment last updated on November 26, 2018 at 12:27 Hours UTC

Choose a reason for hiding this comment

tomneep commented Nov 23, 2018

jreback commented Nov 23, 2018

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

jreback commented Nov 26, 2018

tomneep commented Aug 13, 2018 •

edited by jreback

Loading

codecov bot commented Aug 13, 2018 •

edited

Loading

pep8speaks commented Nov 23, 2018 •

edited

Loading