pycqa / modernize Goto Github PK

and optionally also

• isinstance(_, basestring) to isinstance(_, six.string_types)
• unicode to six.text_type
• str to six.binary_type

(moved from mitsuhiko/python-modernize#18 )

python-modernize currently makes this change:

-        for i in xrange(N):
+        for i in range(N):

The resulting code is inefficient on Python 2.x for large N. I would prefer just to add:

from six.moves import xrange

(moved from mitsuhiko/python-modernize#16 )

While trying to port pylint I have run into itertools.izip and cStringIO as things that need to get moved. It also happens that six.moves handles both cases. Any interest in trying to add a fixer that can work with all renames that six.moves supports?

https://pypi.python.org/pypi?name=modernize&version=0.3&:action=display_pkginfo

Download-url: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN

This would guarantee cross-version semantics of the function itself along with returning the same type (file vs. _io.TextIOWrapper).

    d.addCallback(lambda (rc, out, err), fn=fn:
                  self.failUnlessEqual(out, fn * 1000))

I would have expected this to be changed to something like

    d.addCallback(lambda rc_and_out_and_err, fn=fn:
                  self.failUnlessEqual(rc_and_out_and_err[1], fn * 1000))

(moved from mitsuhiko/python-modernize#23 )

+from __future__ import print_function
 #!/usr/bin/env python

Imports should only be added after any initial lines that are either blank or start with #. That would account for copyright headers etc., as well as shebang lines.

(moved from mitsuhiko/python-modernize#21 )

By default at present, unicode literals are wrapped in a call to six.u, so u'foo' -> six.u('foo').

The --compat-unicode flag leaves unicode literals untouched. The u'foo' syntax is valid from Python 3.3, released in September 2012. I think that projects that haven't yet ported to Python 3 are unlikely to want to support Python 3.2 or older, so I would make the compat-unicode behaviour the default, and add a --six-unicode flag to get the current behaviour.

Basically just replace / with //, under the assumption that integer division was intended. This makes the code clean for DeprecationWarning: classic int division when run under python -3.

(moved from mitsuhiko/python-modernize#22 )

Since fix_metaclass was originally written, six.with_metaclass() has been changed to accept multiple base classes. That would lead to much cleaner output. Compare the current solution:

class Foo(six.with_metaclass(Meta, type('NewBase', (Bar, Spam), {}))):
    pass

to what six would allow for today:

class Foo(six.with_metaclass(Meta, Bar, Spam)):
    pass

While a Python 2.6 and higher fix, it is still legitimate and useful (I just had to go through and do that transformation manually).

Currently python-modernize strips L suffixes off numeric literals. This is dubious because presumably there was a reason for specifying that the value should be long on Python 2.x. (For most arithmetic operations it doesn't matter, but it does affect type testing.)

six doesn't seem to have a function to do a long coercion, although one can easily be defined:

L = int if six.PY3 else long

and then change, e.g. 0xFFFFFFFFL to L(0xFFFFFFFF).

(moved from mitsuhiko/python-modernize#19 )

It would be useful if the README file showed some basic usage examples. I say this as it seems unclear that there is a command "python-modernise" which gets installed and has the same syntax as "2to3".

(moved from mitsuhiko/python-modernize#29 opened by @kevcampb )

@brettcannon has been doing a lot of work improving existing code and adding new fixes. We should think about making a new release so people can use these changes.

I've renamed the milestone from 0.3.1 to 0.4, because it's definitely more than a bugfix release.

To properly match Python 3 import semantics, every import within a package should have from __future__ import absolute_import. lib2to3.fixes.fix_import does a reasonable thing to syntactically make an import use explicit relative imports, but that doesn't guarantee future imports will do the right thing if someone forgets to run the fixer again.

Maybe make an optional fixer that adds the __future__ statement for any code that has an import statement? Another option is to extend lib2to3.fixes.fix_import to add the __future__ statement after calling super().transform() (it also would also make the fixer a little more performant as it uses the __future__ import as a short-circuit). Personally I prefer the latter option for being more thorough.

There are various future_builtins not being used, e.g. fix_map. Using future_builtins would also make fix_zip work when --no-six is specified.

If one has:

    foo(u'.........'
        u'.........')

it gets converted to:

    foo(six.u('........')
        six.u('........'))

that obviously does not work, better would be:

    foo(six.u('.........'
              '.........'))

(moved from mitsuhiko/python-modernize#9 opened by @ThomasWaldmann)

Then one does not need to change this manually after running python-modernize.

(moved from mitsuhiko/python-modernize#10 opened by @ThomasWaldmann)

We should end up with fixers that cover every situation that makes sense, whether listed as an issue or already committed. There is also a selfish reason for doing this as anything not covered by a fixer will need to be documented for the Python porting HOWTO when I rewrite to heavily orient it towards using python-modernize.

That way it guarantees that even calls to file() will get the from io import open fix.

The link to the LICENSE file broke rendering of the README rst on PyPI. I fixed it by hand for this release, but we should work out something better. Maybe just including LICENSE in the tarball is enough. Otherwise it may be easiest just to make it not a link.

People in some pre-commit hook may want to run python-modernize over changed code and reject the commit if it would lead to an actual change. This would allow a code base to be ported and then force developers to actually code in a Python 2/3 compatible way instead of perpetually relying on tools to do it for them.

E.g.

from urlparse import ParseResult

=>

from six.moves.urllib.parse import ParseResult

and

import urlparse
urlparse.ParseResult

=>

import six.moves.urllib.parse as urlparse
urlparse.ParseResult

The idea is that regardless of what str/bytes strategy is chosen, you still need to replace calls to the unicode() type since unicode doesn't exist in Python 3. six.text_type() acts as a replacement since it is unicode() in Python 2 and str() in Python 3.

#38 implements this fixer.

Our current test coverage is pretty poor. There should be tests covering all fixers, including corner cases, and also covering command-line argument handling.

I added support for test coverage reporting via coverage.io: https://coveralls.io/r/python-modernize/python-modernize?branch=master

As mentioned in #31 , there is a desire to document all of the fixers. Probably want to have opt-out vs. opt-in, description of what exactly is changed (e.g. fix_range changes both range() and xrange() but only in calling contexts), and maybe a link to appropriate section of http://python3porting.com/differences.html for reference.

There are two possible parts to such a fixer. One is fixing calls to file() which should probably change to io.open() (or open() depending on the outcome of #42 ). The other part is the use of file for type checks, which requires using io.IOBase, but only if io.open() is used instead of open().

For now it's sufficient for this to show the differences between releases 0.2 and 0.3.

E.g. instead of

-        def _got_root((root, path)):
+        def _got_root(xxx_todo_changeme):
+            (root, path) = xxx_todo_changeme

use

-        def _got_root((root, path)):
+        def _got_root(root_and_path):
+            (root, path) = root_and_path

Also, the X_and_Y convention could be used in cases like this:

-        d.addCallback(lambda (root, path):
-                      self._get_or_create_directories(root, path, metadata))
+        d.addCallback(lambda root_path:
+                      self._get_or_create_directories(root_path[0], root_path[1], metadata))

Here the generated name root_path is inappropriate because that would mean something like "path of the root", as opposed to a (root, path) pair. root_and_path would be better.

(moved from mitsuhiko/python-modernize#15 )

Wrapping dict.items(), et. al. as list(dict.items()) would deal with the case where people actually want the list of items (e.g. A.items() + B.items()). If there is a worry about the generic method names could the fixer be at least opt-in? At least with Pylint I could use this.

Other than the future import, is there any reason to have our own fixer? If not then following the fix_range solution of subclassing and adding an import statement should be enough to keep equivalent semantics and heavily cut down on code we need to maintain.

$ cat test.py
u"artificial example"
import sys

$ python-modernize --six-unicode test.py
[...]
@@ -1,2 +1,3 @@
-u"artificial example"
+six.u("artificial example")
 import sys
+import six

which doesn't work because the import of six is after the first use.

(moved from mitsuhiko/python-modernize#20 )

#58

What I'm thinking of is something like --compat=2.6 such that things like fix_next use the next built-in instead of six.advance_iterator so that the translated code is a bit more idiomatic and less noisy. This also allows for things such as using future_builtins for map, etc. instead of worrying about breaking Python 2.5 code if the user only cares about e.g. Python 2.7 compatibility.

If the project owners are amenable to the idea I'm happy to create a pull request to implement this. I could create separate fixers and have logic in libmodernize.main to select the right fixers based on --compat/--no-six or have all of the selection logic in the fixers themselves and expose the requirements in libmodernize.__init__.

Since the tests directory isn't installed, it's not easy to run tests on the installed version.

If it is installed then it should probably be in libmodernize.tests rather than a toplevel module, to avoid module name clashes.

E.g. [c]StringIO.StringIO should be changed to six.BytesIO.

(moved from mitsuhiko/python-modernize#17 )

As discussed in #61 , for maintenance reasons along with minimizing the possibility of not covering all reasonable argument possibilities that Python supports -- at the expense of possibly over-applying when a name of a function or method just happens to match -- we should simplify the matching patterns for function calls.

The typical pattern should probably be trailer< '(' ')' > when no argument is ever allowed, trailer< '(' any+ ')' > when at least one argument needs to be there, and probably something like trailer< '(' [any+] ')' > when 0 or more (the last pattern is untested).

Looks like the comment was pasted from libmodernize/fixes/fix_next.py and not updated.

(moved from mitsuhiko/python-modernize#25 )

raw_input should become six.moves.input. Ties into #34.

I.e. modernize.__version__ == "0.3".

This is useful to check which version you have installed.

Since most uses of basestring are for isinstance() calls, could you use (unicode, str) if hasattr(__builtins__, 'unicode') else str instead of basestring for isinstance/issubclass calls?

2to3 allows you to select fixes using short names, e.g. sys_exc instead of lib2to3.fixes.fix_sys_exc. Modernize should ensure that the same thing works, so a user wanting to apply a single fix doesn't have to type out the full path.

Without knowing exactly what fixers are allowed (not) to do, I'm starting to see some disagreement of whether an approach is the best solution or not. Ignoring the syntax-only fixers which are have zero controversy -- e.g. except Exception as exc: ... -- I'm not sure what guidelines there are which ones overrule what. Some questions I know I either have or have seen in other issues:

• Should all (at least default) fixers be idempotent (issues #44 and #52 )?
• Should fixers be allowed to lightly break code, even at the price of idempotency (issue #53 )?
• Should the Python 2 way of doing things or the Python 3 was be preferred?

Take the whole range()/xrange() issue as an example. If being idempotent is the the most important thing while also allowing Python 3 semantics, then you will want:

from six.moves import range
range(1)  # was range(1)
range(2)  # was xrange(2)

If you don't want to break any code but still want idempotency over Python 3 features, then you will want:

list(range(1))  # was range(1)
list(range(2))  # was xrange(2)

If you don't want to break any code while allowing Python 3 features but don't care about idempotency:

from six.moves import range
list(range(1)) # was range(1)
range(2)  # was range(2)

Three possible outcomes all based on what priorities you place on the resulting code and what your plans are for the future of it.

For me personally, I want to see this project help other projects get their code modernized as much as possible: that means I prefer an approach which is idempotent so that they can benefit from newer fixers and accidental changes that will hurt Python 3 compatibility, but with a willing cost of lightly breaking code (e.g. triggering a TypeError when you are doing from six.moves import range; range(1) + range(2) is acceptable to me). IOW getting people to code using Python 3 features is paramount to me while idempotency is important for easier tooling.

One approach we can take is for any fixer which cannot be totally unambiguous and won't break code while stilling being idempotent then it is an optional/opt-in fixer -- do we have a preferred term for this? -- and that is the end of that. If people want help with range()/xrange() we won't make the choice for them and they will simply have to choose a fixer. That would mean fix_filter, fix_map, fix_range, and fix_zip would all become optional as they all change the semantics to some degree or another. Making them optional is fine by me, but as stated previously my opinion is that what they do now -- sans fix_range stopping trying to wrap range() in a list() call -- is acceptable.

fix_map and fix_zip both add a from six.moves import, without changing the code at all. This means that code is getting the iterator versions of fix and map consistently across Python 2 and 3, but it can break code which relied on getting the list version. I think we should improve this.

I think the fixers should work like this:

for i in map(...):
    ...
# Converts to:
from six.moves import map
for i in map(...):
   ...

a = map(...)  # anything other than iterating over map in a loop
# Converts to:
from six.moves import map
a = list(map(...))

Now that libmodernize.fixes.fix_range is being used instead of lib2to3.fixes.fix_xrange, there is the problem that the fixer is no longer idempotent, i.e. list(range(1)) will become list(list(range(1))) and from six.moves import range; range(1) becomes from six.moves import range; list(range(1)).

Possible solutions are:

1. Do the necessary bookkeeping for both range and xrange (I suspect range is doable with some not rule in the pattern, but xrange might be a pain)
2. Only fix xrange and assume anyone using range to make a list can do the fix themselves
3. Go back to using lib2to3.fixes.fix_xrange which turns xrange into range