RFC0017-Domain-Specific-Language-Specifications about powershell-rfc

[rjmholt]

@felixfbecker this RFC is fairly old now and I think opinions may have changed since it was written -- especially after one or two attempts at implementing it.

Personally, I would like to implement DSL support that would enhance existing PowerShell DSLs with minimal changes (like an attribute in the function definition). Whether or not people agree that DSLs in PowerShell are a "good thing", they already exist -- so my thinking is that it would be nice to help those existing cases with better syntactic support, rather than a new and obscure loading/semantics pathway.

I've been thinking about a better way to do this recently, but need to get back to my implementation here -- it basically just adds an attribute to functions so that the parser knows you can hang a brace, maybe highlight the function name differently (although VSCode wouldn't be able to support this), and get better completions for the body (last two parts not yet implemented...).

On your specific point on the ps1xml files, the arguments there were:

• I find that XML tends to be (compared to other markup languages like YAML or TOML):
  • prolix
  • tedious to write
  • hard to read
  • surprisingly complicated when people make choices as to what is an attribute and what is a child element
  • semantically obscure (arguably true of all markup languages, but perhaps less for an embedded DSL -- I had to debug a csproj the other day and it took more time than it should have)
• ps1xml specifically is
  • obscure
  • not well documented
  • lacking in completion/checking tooling
  • alternately implemented with a C# API
• so a DSL for ps1xml should give you completions and possibly even checking in an interactive scenario, while always leveraging the fact that the user just needs to know PowerShell -- it was a showcase scenario for DSLs
• with all that in mind, it's fairly straightforward to write an "extended builder pattern" DSL for the C# ps1xml types as PowerShell functions that gives you the PowerShell-embedding part of the ps1xml, after which you would just want the actual feature reasons for the DSL, namely the completions and checking. @KevinMarquette has a good set of posts on this

But I don't think ps1xml is quite motivating enough -- the existing DSLs I think are more important. I would like highlighting, brace-hanging and keyword completions in Pester and InvokeBuild.

Another consideration that @rkeithhill brought up for Plaster in PowerShellOrg/Plaster#334 (comment) was that DSLs may execute arbitrary code and that isn't always wanted. PowerShell already offers Constrained Language Mode, so maybe that's the way to go, but I'm not sure. Watching @KevinMarquette's talk at PowerShell Summit, it was clear that the advantage of the DSL is that you can generate large numbers of entries with expressions -- so still being able to do that in a secure way might be the sweet spot.

[SteveL-MSFT]

PowerShell attributes don't have the PS prefix, we should be consistent here (and also match the PowerShell script syntax for DSLs).

It would be interesting to see more complete examples for the example motivating DSLs to validate the proposed syntax is sufficient and not unnecessarily complex for real-world usage. Perhaps these would be part of the test cases.

[SteveL-MSFT]

@DerpMcDerp a psd1 file is essentially a hashtable and one limitation is that it's really just text until runtime so you don't get things like intellisense so discovery of keywords is a problem without relying on external documentation or utilities like new-modulemanifest.

macros have some benefits (and with great power comes great reponsibility kind of thing), but also has similar limitations in terms of discovery which is a driving motivation for this RFC.

regarding ps1xml, it also has the same problem with discovery.

[DerpMcDerp]

@SteveL-MSFT You modify PowerShell to recognize a special hashtable schema comment annotation that makes IDEs do fancy-pants IDE stuff on it.

[rjmholt]

I've submitted a new PR for a revision of this RFC. Hoping to simplify the specification and provide a proposal that's more straightforward and consistent with existing PowerShell features (namely Cmdlet instantiation).

[SteveL-MSFT]

@PowerShell/powershell-committee has reviewed this and agreed to move it to Experimental stage (RFC has been moved to appropriate folder)

[SteveL-MSFT]

@ebekker unfortunately we never merged the prototype code. @daxian-dbw is working on getting a DSL implementation into PSCore6, but it's not a 6.0.0 deliverable. You can clone and build the branch that @rjmholt worked on if you want to play with it PowerShell/PowerShell#3169

[SteveL-MSFT]

This was accepted as Experimental, if anyone has questions/concerns, please open individual issues against this RFC

[rjmholt]

@SteveL-MSFT
Are the PowerShell attributes like those documented here? It would be nice to find something consistent with current usages, but without occupying too general a name in the (conceptual) attribute namespace.

I'd like to provide a more complete example in the RFC before it's out of draft, but one succinct enough to keep the RFC readable. I very much expect that more complex examples will form part of test cases, ideally drawing on pre-existing examples of DSLs in PowerShell.

[DerpMcDerp]

This proposal seems to be conflating 2 unrelated things: 1. getting rid of all the XML configuration PowerShell uses and 2. adding custom syntax to PowerShell.

In the former case, other people have suggested replacing the configuration files with JSON. My personal preference is for PowerShell's data language (e.g. the .psd1 file).

In the latter case, a macro language like the one proposed for javascript seems like a better solution than some finicky DSL.

[rjmholt]

@DerpMcDerp , to add to @SteveL-MSFT 's comment, while macros could be a very useful tool for PowerShell, they tend to be very large undertakings and the languages that arguably do them well (e.g. Racket, Rust, C++) implement them as core features early on and leverage them heavily thereafter. Macro implementations added later to established languages can be more trouble than they're worth.

In PowerShell's case, the DSL functionality is probably aiming for a smaller audience and only seeks to provide a way for users to make safe additions to the PowerShell syntax in small, well-defined ways (which they can implement their own checks for using the API), rather than requiring a macro framework to validate arbitrary changes to the AST.

I agree there is some conflation of XML and syntax customisation, although I think it's in the RFC (and will try and clarify it) rather than the proposed feature itself. Really a ps1xml file contains a kind of declarative language -- adding DSL creation functionality to PowerShell also gives us the opportunity to bring that declarative language and those like it into PowerShell proper.

[ebekker]

On the community call, it was confirmed that since this is in experimental phase, there is proof-of-concept or prototype code that implements this RFC. Is it possible for mere mortals to some how get access to this code, for example, some special build of PS6?

[felixfbecker]

I am somewhat concerned about a feature that makes DSL easier in general. You can find a lot of voices on the internet on the disadvantages of DSLs, e.g. Death by a thousand DSLs. I don't want PowerShell to become like Scala where you can get into a random code base and not understand a single line of code because there is so much metaprogramming and DSLs that every codebase feels like a completely different language.

BDD test syntax like in Pester is wide-spread and implemented without DSL support as simple functions that register tests to run in other languages too, e.g. mocha in JavaScript.

Types.ps1xml files can already be expressed in a type-safe way in PowerShell with Update-TypeData.

In these two cases, the mentioned benefits are already achieved without special DSL support.

Format.ps1xml files define UI, and it is common to define UI in XML (see XAML and JSX), so not sure if having a PowerShell DSL is even desirable. In this case, the benefits listed (except "less XML", which is a somewhat recursive point: "we need less XML because we need less XML") can also be achieved easier in different ways without changing the whole definition mechanism, see PowerShell/PowerShell#7749 for a proposal on this.

I think before proceeding further, the RFC should do a better job of explaining why a DSL would be better than the status quo for the mentioned cases.

[SteveL-MSFT]

@felixfbecker I think that is fair feedback. Thinking about this just now, if we support an opening brace on the next line if we know there is a positional scriptblock, then perhaps the current use of funtions as keywords is sufficient.