xp-framework / compiler

Compiles future PHP to today's PHP.

XP Compiler

Compiles future PHP to today's PHP.

Usage

After adding the compiler to your project via composer require xp-framework/compiler, it will hook into the class loading chain and compile .php-files on-demand. This keeps the efficient code-save-reload/rerun development process typical for PHP.

Example

The following code uses Hack language, PHP 8.4, PHP 8.3, PHP 8.2, 8.1 and 8.0 features but runs on anything >= PHP 7.4. Builtin features from newer PHP versions are translated to work with the currently executing runtime if necessary.

<?php // In a file "HelloWorld.php"

use lang\Reflection;
use util\Date;
use util\cmd\Console;

#[Author('Timm Friebe')]
#[Permissions(0o777)]
class HelloWorld {
  private const string GREETING= 'Hello';

  public static function main(array<string> $args): void {
    $greet= fn($to, $from) => self::GREETING.' '.$to.' from '.$from;
    $author= Reflection::type(self::class)->annotation(Author::class)->argument(0);

    Console::writeLine(new Date()->toString(), ': ', $greet($args[0] ?? 'World', from: $author));
  }
}

To run this code, use xp -m /path/to/xp/reflection HelloWorld in a terminal window.

Compilation

Compilation can also be performed explicitely by invoking the compiler:

# Compile code and write result to a class file
$ xp compile HelloWorld.php HelloWorld.class.php

# Compile standard input and write to standard output.
$ echo "<?php ..." | xp compile -

# Compile src/main/php and src/test/php to the dist folder.
$ xp compile -o dist src/main/php/ src/test/php/

# Compile src/main/php to the dist.xar archive
$ xp compile -o dist.xar src/main/php/

# Compile src/main/php, do not write output
$ xp compile -n src/main/php/

# Target PHP 7.4 (default target is current PHP version)
$ xp compile -t php:7.4 HelloWorld.php HelloWorld.class.php

# Emit XP meta information (includes lang.ast.emit.php.XpMeta):
$ xp compile -t php:7.4 -a php:xp-meta -o dist src/main/php

The -o and -n options accept multiple input sources following them. The -q option suppresses all diagnostic output except for errors.

Features supported

XP Compiler supports features such as annotations, arrow functions, enums, property type-hints, the null-safe instance operator as well as all PHP 7 and PHP 8 syntax additions. A complete list including examples can be found in our Wiki.

More features

Additional syntax like an is operator, generics or record types can be added by installing compiler plugins from here:

$ composer require xp-lang/php-is-operator
# ...

$ xp compile
Usage: xp compile <in> [<out>]

@FileSystemCL<./vendor/xp-framework/compiler/src/main/php>
lang.ast.emit.PHP74
lang.ast.emit.PHP80
lang.ast.emit.PHP81
lang.ast.emit.PHP82
lang.ast.emit.PHP83 [*]
lang.ast.emit.PHP84
lang.ast.emit.PHP85
lang.ast.syntax.php.Using [*]

@FileSystemCL<./vendor/xp-lang/php-is-operator/src/main/php>
lang.ast.syntax.php.IsOperator

Implementation status

Some features from newer PHP versions as well as Hack language are still missing. The goal, however, is to have all features implemented - with the exception of where Hack's direction conflicts with PHP! An overview can be seen on this Wiki page.

To contribute, open issues and/or pull requests.

See also

• XP Compiler design
• XP RFC #0299: Make XP compiler the Babel of PHP
• XP RFC #0327: Compile-time metaprogramming

compiler's Issues

Fix traits

Call to undefined method lang\ast\emit\PHP71::emituse()

Warning in emitter

lang.ast.Emitter::emit() [line 703 of Emitter.class.php] Invalid argument supplied for foreach()

Compiler error: Call to undefined method lang\ast\emit\PHP72::emitoperator()

This occured when parsing a hack-style lambda:

$authenticate= newinstance(Filter::class, [], [
  'filter' => ($request, $response, $invocation) ==> {
    // ...
  }
]);

I'd expect this to yield a parse error!

Support using

HHVM 3.24 added a new using statement as a step to deprecate destructors:

interface IDisposable {

  /** Called at the end of a using statement */
  public function __dispose(): void;
}

using ($f= new File()) {
  // ...
}  // $f->__dispose() is implicitly called here

See also https://hhvm.com/blog/2018/01/16/hhvm-3.24.html and https://docs.hhvm.com/hack/disposables/introduction

XP Framework has had a with core function for quite a while, which is similar, but builds ontop of lang.Closeable, see xp-framework/rfc#172. This comes with the overhead of a function call, though. If realized in the compiler, this runtime overhead could be omitted,

Create comprehensive documentation

I've started writing an longer overview including short examples @ https://github.com/xp-framework/compiler/wiki

/cc @mikey179

Interop with xp-forge/partial broken

Dynamic traits cannot be resolved, resulting in Cannot instantiate incomplete type com\example\user\lang\partial\Value

instanceof does not resolve class names

#[@test]
public function instanceof_imported_type() {
  $r= $this->run('use util\Date; class <T> {
    public function run() {
      return new Date() instanceof Date;
    }
  }');

  $this->assertTrue($r);
}

Assertion fails.

Warnings for `return;`

Invalid argument supplied for foreach()" in lang\ast\Emitter::emit() (Emitter.class.php, line 774, occured 2 times)

Namespace declaration statement has to be the very first statement

Running the test suite with PHP 8.0.0, RC1 yields the following: Compile error (Namespace declaration statement has to be the very first statement or after any declare call in the script)

This is caused by the emitter yielding a ; after the PHP open tag:

$ cat ns.php
<?php ;namespace test;

echo "Works\n";

$ XP_RT=7.4 xp ns.php
Works

$ XP_RT=8.0 xp ns.php
Uncaught error: Compile error (...)

Ternary operator broken

return $value ? "OK" : "Fail";  // Returns "OK" for $value= false

Support keywords as methods

See https://wiki.php.net/rfc/context_sensitive_lexer:

class Collection extends \ArrayAccess, \Countable, \IteratorAggregate {
 
    public function forEach(callable $callback) {
        //...
    }
 
    public function list() {
        //...
    }
 
    public static function new(array $itens) {
        return new self($itens);
    }
}

Records idea

Inspired by https://www.infoq.com/articles/java-14-feature-spotlight/

records automatically acquire implicit implementations of the canonical constructor (the one whose signature matches the state description), read accessors for each state component (whose names are the same as the component), private final fields for each state component, and state-based implementations of the Object methods equals(), hashCode(), and toString().

Call to a member function children() on array

class Test {
  public function run($arg) {
    return fn() => {
      if ($arg < 10) {
        return $arg + 1;
      } else {
        return $arg;
      }
    };
  }
}

Compile to directory

$ xp compile src/main/php dist

The output files' names will be created from the input files' names stripped of the prefix src/main/php and replace the extension with .class.php.

Comments contain stars

This breaks the usage for XPCLIs.

Example

" * ONE - the game
 *
 * Get points for posting, commenting and liking on ONE! You become a player
 * in this game by joining https://inside.example.org/one/#walls/453
"

Solution

Replace them as follows inside the Tokens class:

// Old
yield 'comment' => [substr($comment, 2, -3), $line];

// New
yield 'comment' => [trim(preg_replace('/\n\s+\* ?/', "\n", substr($comment, 1, -2))), $line];

Index type members by name

For easy access to testing whether a property, constant or method already exists.

Comments missing from generated code

...but they're used for example by XP commands:

$ xp cmd Punktestand -?
Uncaught exception: Exception lang.IndexOutOfBoundsException (Undefined offset: 4)
  at <main>::__error(8, (0x13)'Undefined offset: 4', (0x6a)..., 559, array[1]) [line 559 of XPClass.class.php]
  at lang.XPClass::getComment() [line 66 of CmdRunner.class.php]
  at xp.command.CmdRunner::commandUsage(lang.XPClass{}) [line 128 of AbstractRunner.class.php]
# ...

Line 559 reads:

return $details['class'][DETAIL_COMMENT];

Short closures

Currently, the compiler supports the Hack language variant, see https://docs.hhvm.com/hack/functions/anonymous-functions:

array_map((User $user) ==> $user->id, $users)

However, PHP 7.4 has decided to go a different way, introducing the fn keyword in https://wiki.php.net/rfc/arrow_functions_v2, which makes it:

array_map(fn(User $user) => $user->id, $users)

The XP Compiler should adopt the latter.

Support generics

Example

// Declaration
class Vector<T> {
  public function add(T $t): self { ... }
  public function get(int $i): T { ... }
}

// Instantiation
$vector= new Vector<string>();

See also

xp-framework/rfc#106
xp-framework/rfc#193
https://wiki.php.net/rfc/generics
https://docs.hhvm.com/hack/generics/introduction

Null Coalescing Assignment Operator

https://wiki.php.net/rfc/null_coalesce_equal_operator has been implemented in PHP 7.4

Resolved paths not absolute

# Code loaded through standard filesystem classloader
$ xp -w 'return array_pop(get_included_files())'
C:\...\src\main\php\xp\runtime\Modules.class.php

# Code compiled on demand
$ xp -w 'new Test(); return array_pop(get_included_files())'
Test.php

This breaks tools that expect the paths to be absolute.

Enums broken

Static intializers are not run when loading via CompilingClassLoader.

Preliminary PHP 8 support

See https://travis-ci.org/xp-framework/compiler/jobs/545844314

Uncaught exception: Exception lang.IllegalArgumentException (
  XP Compiler does not support PHP.8.0.0-dev yet
)

Segmentation fault with PHP 7.3.0-dev

See https://travis-ci.org/xp-framework/compiler/jobs/389491676

Deprecate Hack language style arrow functions

Emit deprecation warnings for Hack language style arrow functions and compact methods using ==>, instead advocate the use of PHP 7.4 with the fn keyword.

By using warnings, production code will continue to work; while unit tests will fail as seen below:

Syntax error for parameters called "function"

class Test {
  public function map($function) {
    // ...
  }
}

Causes the error Expected "(", have ")"

Test suite failure on HHVM 3.25

lang.ast.unittest.emit.VarargsTest::list_of

Parameter $args is variadic and has a type constraint (HH\string); variadic params with type constraints are not supported in non-Hack files

Member types missing for constructor argument promotion

#[@test]
public function type_information() {
  $t= $this->type('class <T> {
    public function __construct(private int $id) {
    }
  }');
  $this->assertEquals(Primitive::$INT, $t->getField('id')->getType());
}

Check generator return feasibility in PHP 5

In PHP 5, a generator could not return a value: doing so would result in a compile error. An empty return statement was valid syntax within a generator and it would terminate the generator. Since PHP 7.0, a generator can return values, which can be retrieved using Generator::getReturn().

See also http://php.net/manual/en/generator.getreturn.php

Support list() Reference Assignment

https://wiki.php.net/rfc/list_reference_assignment - implemented in PHP 7.3

String parsing of escapes broken

$ cat escape.php
<?php

print('\\\'');

# Executes in PHP
$ cat escape.php | xp -e
\'

# But doesn't in XP compiler:
$ xp compile escape.php
<?php Uncaught exception: Exception lang.FormatException (Unclosed string literal starting at line 3)
  at lang.ast.Tokens::getIterator() [line 0 of StackTraceElement.class.php]
  at Generator::next() [line 1411 of Parse.class.php]
  at lang.ast.Parse::advance() [line 1277 of Parse.class.php]
  at lang.ast.Parse::expression(0) [line 1307 of Parse.class.php]
  at lang.ast.Parse::statement() [line 1286 of Parse.class.php]
  at lang.ast.Parse::top() [line 796 of Emitter.class.php]
  at lang.ast.Emitter::emit(Generator{}) [line 79 of CompileRunner.class.php]
  at xp.compiler.CompileRunner::main(array[1]) [line 374 of class-main.php]

Adjust to new PHP annotation syntax

Use #[...] instead of << >>, see https://wiki.php.net/rfc/shorter_attribute_syntax

• Support both in the first step in a patch level release
• Add an E_DEPRECATED warning to << >> syntax in order to make unittests fail but production code work in next minor release
• Remove support for << >> in next major release

Comments are not escaped

If a comment contains a single quote ', a syntax error occurs.

Move parameter annotations to parameter

public function __construct(<<inject>> Database $repo) {
  // ...
}

See also xp-framework/rfc#218

Throw expressions

In a nutshell

return $user ?? throw new IllegalStateException('No such user');

Would be allowed for the ternary operator, the null-coalesing operator and inside of compact functions.

See also

• https://blogs.msdn.microsoft.com/dotnet/2017/03/09/new-features-in-c-7-0/
• https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/throw
• https://github.com/tc39/proposal-throw-expressions

Conditional catch

Idea:

try {
  …
} catch (LDAPException $e) if (83 === $e->getCode()) {
  …
}

Inspired by C#'s catch … when form:

catch (ArgumentException e) when (e.ParamName == "…")  
{  
}  

(see https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/try-catch)

...and JavaScript's non-standard (and no longer supported) if:

try {
    … 
} catch (e if e instanceof TypeError) {
    // statements to handle TypeError exceptions
}

(see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/try...catch)

Enum support

Add syntactic support for enums

enum WeekDay {
  MON(1), TUE, WED, THU, FRI, SAT, SUN;
}

See:

• https://wiki.php.net/rfc/enum PHP RFC
• xp-framework/rfc#132 XP RFC (merged October 2011)
• https://docs.hhvm.com/hack/enums/introduction Hack Language
• http://captainkuro.com/php/enums-in-php/ Overview of different solutions

Support "mixed"

...emitting as var for XP reflection. Only useful for consistency purposes by itself, because you can just as well omit the typehint alltogether from a technical point of view, but important for arrays:

public function headers(): array<string, mixed> {
  // The equivalent of putting `@return [:var]` into the apidocs
}

See https://wiki.php.net/rfc/mixed-typehint and https://docs.hhvm.com/hack/types/type-system#mixed

Optimize runtime performance by including annotations in code

Generate xp::$meta["f.q.c.N"]= ... beneath each type.

Integrate throw expression

lang.ast.syntax.php.ThrowExpression can be integrated as throw expressions are now part of PHP 8, see https://wiki.php.net/rfc/throw_expression

$ XP_RT=8.0 xp -w 'return $args[1] ?? throw new \lang\IllegalArgumentException("Missing arg")'
Uncaught exception: Exception lang.IllegalArgumentException (Missing arg)
  at <main>::include() [line 154 of Code.class.php]
  at xp.runtime.Code::run(array[1]) [line 30 of Dump.class.php]
  at xp.runtime.Dump::main(array[1]) [line 381 of class-main.php]

"xp compile" installation

Skipped installation of bin bin/xp.xp-lang.compiler.compile for package xp-framework/compiler: file not found in package

Support named arguments

https://wiki.php.net/rfc/named_params - implemented in PHP 8.0:

// Using positional arguments:
array_fill(0, 100, 50);
 
// Using named arguments:
array_fill(start_key: 0, num: 100, val: 50)

• Syntactic support
• Emit argument names in PHP 8
• Argument names erasure for PHP 7 (initial support)
• Check feasibility of full support in PHP 7 (at compile time or at runtime?)

Numeric Literal Separator

$threshold = 1_000_000_000;  // a billion!
$testValue = ‪107_925_284.88; // scale is hundreds of millions
$discount = 135_00;          // $135, stored as Cents

See https://wiki.php.net/rfc/numeric_literal_separator

Nullable cast (?int)

$number = 123; 
(int) $number => 123; 
(?int) $number => 123;

$string = '123'; 
(int) $string => 123; 
(?int) $string => 123;

$null = null; 
(int) $null => 0 
(?int) $null => null

See https://externals.io/message/102997

Class not found

If a class is loaded from a class path during class path setup, and CompilingClassLoader is already registered, it may not find all classes correctly.

FileSystemCL<./vendor/xp-framework/core/src/main/php>
FileSystemCL<./vendor/xp-framework/core/src/main/resources>
FileSystemCL<./vendor/xp-framework/compiler/src/main/php>
CompilingCL<lang.ast.emit.PHP71>
FileSystemCL<./vendor/xp-forge/web/src/main/php>
FileSystemCL<./src/main/php>

If a class is loaded from ./vendor/xp-forge/web/src/main/php during startup, this will initialize the loaders member. Classes requested from ./src/main/php will then no longer be found.

Annotations in package

....are still parsed from code, rather than being loaded from xp::$meta. Reason ist this is seeded wrongly for namespaces.

Support null-safe instance operator

$name= $user?->name();  // will be null if $user is null

https://docs.hhvm.com/hack/operators/null-safe
https://wiki.php.net/rfc/nullsafe_calls (Draft)

Support unicode escape sequences

Only for PHP 5.6, see https://wiki.php.net/rfc/unicode_escape

$str= "\u{1F602}"; // 😂

Arrow functions with blocks

// Works today
$f= ($req, $res) ==> $res->answer(202);

// New
$f= ($req, $res) ==> {
  // ...
  throw new Error(404, 'Not found');
};

Consistent with https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions

Typed properties @ PHP 7.4

https://wiki.php.net/rfc/typed_properties_v2 has been implemented. A PHP 7.4 emitter could now emit property types.