py3to2 2008.11.22

################################################################################
py3to2 is a python2.6 interpreter w/ extended python3.0 opcodes, allowing it to
natively run python3.0 scripts. it should b fully backwards-compatible w/
cpython2.6 & its extensions.

the intended purpose is to allow developers to migrate python2.6 scripts to
python3.0 while retaining backwards compatibility w/ existing extension modules.

AUTHOR:
kai zhu
kaizhu@ugcs.caltech.edu

REQUIREMENTS:
- posix/unix os (Windows currently unsupported)
- w/ python2.6 & python3.0 installed

INSTALL
$ python2.6 setup.py build
$ python2.6 setup.py install
$ python2.6 setup.py dev --quicktest

the above will build & install 3 files:
- extended python2.6 interpreter: bin/py3to2
- initialization script: lib/python2.6/site-packages/py3to2_init.py
- python3.0 bytecode compiler: lib/python2.6/site-packages/py3to2.py

MAGIC
simply add the MAGIC LINE:

from __future__ import py3k_syntax

to make py3to2 aware that a script is using python3.0 syntax

PSEUDOMETHOD:
py3to2 supports ".." syntax notation
please goto: http://pypi.python.org/pypi/pseudomethod
for more details about this feature

API:
try help(py3to2) ^_-

py3to2 module:
- class codetree - mutable codeobj & disassembler/assembler/debugger
- class compiler - compiling tools
- python3.0 wrappers:
- py3k_compile() - compile python3.0 src
- py3k_eval() - eval py3thon3.0 src
- py3k_exec() - exec python3.0 src

USAGE:
start up the py3to2 interpreter by typing "py3to2" in ur shell:
$ py3to2

Python 2.6.py3to2 (r26:66714, Nov 18 2008, 00:56:43)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

try out this simple python3.0 script:
################################################################
# PEP3132 Extended Iterable Unpacking
# copy this to file test_pep3132.py

from __future__ import py3k_syntax

a,*b = 1,2,3
assert a == 1 and b == [2,3]
print(a,b)
################################################################
>>>
>>> import test_pep3132
created...
py3k server starting...
...py3k server started w/...

1 [2, 3]

here's another python3.0 script using scipy (python2.6) extension module:
################################################################
# u must have scipy installed for this script to work
# copy this to file test_pep3132_scipy.py

from __future__ import py3k_syntax

import scipy
a,*b = scipy.array([1,2,3])
assert a == 1 and b == [2,3]
print(a,b)
################################################################
>>>
>>> import test_pep3132_scipy
1 [2, 3]

another simple, but more thorough test script, test_py3k,
is included w/ this distribution:
>>>
>>> import test_py3k
testing PEP3102 Keyword-Only Arguments
testing PEP3104 Access to Names in Outer Scopes
testing PEP3105 Make print a function
testing PEP3107 Function Annotations
testing PEP3112 Bytes literals in Python 3000
testing PEP3113 Removal of Tuple Parameter Unpacking
testing PEP3114 Renaming iterator.next() to .__next__()
testing PEP3115 Metaclasses in Python 3000
testing PEP3127 Integer Literal Support and Syntax
testing PEP3129 Class Decorators
testing PEP3132 Extended Iterable Unpacking
testing PEP3135 New Super
testing pseudomethod example 0
testing pseudomethod example 1
testing pseudomethod example 2
testing pseudomethod example 3
testing numpy example

FEATURES:
PEP3102 Keyword-Only Arguments
PEP3104 Access to Names in Outer Scopes
PEP3105 Make print a function
PEP3107 Function Annotations
PEP3111 Simple input built-in in Python 3000
PEP3112 Bytes literals in Python 3000
PEP3113 Removal of Tuple Parameter Unpacking
PEP3114 Renaming iterator.next() to .__next__()
PEP3115 Metaclasses in Python 3000
PEP3127 Integer Literal Support and Syntax
PEP3129 Class Decorators
PEP3132 Extended Iterable Unpacking
PEP3135 New Super

LIMITATIONS (FEATURES NOT FULLY SUPPORTED):
from a migration standpoint, py3to2 is almost feature complete in terms of
python3.0's language syntax, except for:
- unicode support (str vs bytes). future support for utf8 is pending...

language issue aside, python3.0 scripts will still behave differently b/c of
internal differences between python2.6 & python3.0:
- exception handling. py3to2 implements python3.0 syntax for raising &
catching exceptions. but the underlying behavior is still python2.6
- builtin functions / types. a few of these have become different beasts
under python3.0

################################################################################
MECHANISM

py3to2 has 3 components:
- py3to2
python interpreter. can evaluate python2.6 bytecode containing additional
python3.0 opcode instructions

- py3to2_init.py
initialization script. sets up import hook for recognizing python3.0 scripts

- py3to2.py
bytecode compiler. the compile process takes 2 steps:
- a persistent python3.0 process is created for compiling scripts into
python3.0 code
- py3to2.py then converts the code from python3.0 to python2.6 format

################################################################################
MANIFEST
./patch/ - patched files
./py3to2.diff - summary of patches (maybe out-of-date)

PYTHON2.6 COMPATIBILITY TEST
output from amd opteron x86_64 machine on redhat linux (3.4.6-10):
$ python setup.py dev --maketest
...
324 tests OK.
36 tests skipped:
test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
test_cd test_cl test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_dl test_gdbm test_gl test_imageop test_imgfile test_kqueue
test_linuxaudiodev test_macos test_macostools test_normalization
test_ossaudiodev test_pep277 test_py3kwarn test_scriptpackages
test_socketserver test_startfile test_sunaudiodev test_timeout
test_urllib2net test_urllibnet test_winreg test_winsound
test_zipfile64
1 skip unexpected on linux2:
test_gdbm

RECENT CHANGES:
moved pseudomethod syntax handling to py3k server
added more checks during setup
added more documentation
backported patch r67299 fixing an issue w/ super()
cleaned up py3to2.compiler class
20081120
fixed package importing bug - py3to2 failed to import foo.bar
20081119
created self-installing distutils distribution
20081019
ported to python-2.6
consolidate & simplify patches to 1 file: ceval.c
created extension module builtins_py3k
revamped import hook again
removed unicode support & restrict source code to ascii-only
20080727
revampled import hook
20080911
consolidate patches to 2 files: bltinmodule.c & ceval.c
20080828
add kwonlyargcount 'attr' to codeobj
add __annotations__ & __kwdefaults__ attr to funcobj
add __pseudomethod__ feature to baseobj
20080819
pure python import hook - removed magic comment & use magic path instead
revamped str & bytes handling
revamped py3k .pyc file handling
20080802
pep3135 New Super
20080717
pep3107 Function Annotations
pep3120 Using UTF-8 as the default source encoding
pep3131 Supporting Non-ASCII Identifiers
20080713
import / reload works transparently on py3k scripts using a magic comment
added pep3102 Keyword-Only Arguments
20080709 added a py3k preparser
20080702
rewrote py3k server's pipe io. implemented partial bytearray & bytes class.
wrote a few simple tests
20080630
__build_class__ function to bltmodule.c. tested class decorators to b working.
################################################################################