puppetlabs / pdk

The shortest path to better modules: Puppet Development Kit; Download:

pdk

• Installation
• Basic usage
• Experimental features
• Module compatibility
• Contributing
• Contact Information

The Puppet Development Kit (PDK) includes key Puppet code development and testing tools for Linux, Windows, and OS X workstations, so you can install one package with the tools you need to create and validate new modules.

PDK includes testing tools, a complete module skeleton, and command line tools to help you create, validate, and run tests on Puppet modules. PDK also includes all dependencies needed for its use.

PDK includes the following tools:

Tool	Description	Owned by Puppet
facterdb	A gem that contains facts for Operating Systems.	N
metadata-json-lint	Validates and lints metadata.json files in modules against Puppet module metadata style guidelines.	N
pdk	Tool to generate and test modules and module content, such as classes, from the command line.	Y
puppet-lint	Checks your Puppet code against the recommendations in the Puppet language style guide.	Y
puppet-syntax	Checks for correct syntax in Puppet manifests, templates, and Hiera YAML.	N
puppetlabs_spec_helper	Provides classes, methods, and Rake tasks to help with spec testing Puppet code.	Y
rspec-puppet	Tests the behavior of Puppet when it compiles your manifests into a catalog of Puppet resources.	Y
rspec-puppet-facts	Adds support for running rspec-puppet tests against the facts for your supported operating systems.	N

Installation

Download and install the newest package matching your platform from the download page. If you are using Windows, remember to close any open PowerShell windows.

For complete installation information, see the PDK documentation.

Basic usage

PDK can generate modules and classes, validate module metadata, style, and syntax, and run unit tests. This README contains very basic usage information---for complete usage information, see the PDK documentation.

Generate a module

Generates the basic components of a module and set up an infrastructure for testing it with PDK.

1. Run the pdk new module command, specifying the name of the new module:

pdk new module my_module

This command asks a series of metadata questions and then generates the basic components of a new module.

Generate a class

To generate a class in your module, use the pdk new class command, specifying the name of your new class. To generate the main class of the module, which is defined in an init.pp file, give the class the same name as the module.

1. From the command line, in your module's directory, run:

pdk new class class_name

PDK creates the new class manifest and a test file (as class_name_spec.rb) in your module's /spec/classes directory.

Generate a defined type

To generate a defined type in your module, use the pdk new defined_type command, specifying the name of your new defined type.

1. From the command line, in your module's directory, run:

pdk new defined_type defined_type_name

PDK creates the new defined_type manifest and a test file (as defined_type_name_spec.rb) in your module's /spec/defines directory.

Generate a task

To generate a task in your module, use the pdk new task command, specifying the name of your new task.

1. From the command line, in your module's directory, run:

pdk new task task_name

PDK creates the new task file and metadata.

Validating a module

PDK can validate the well-formedness of the module and style and syntax of its files.

1. In the module's directory, run:

pdk validate

This command validates the metadata, Puppet code syntax and style, and Ruby syntax and style for the entire module.

Run unit tests

PDK's default template sets up rspec for Ruby-level unit testing, and rspec-puppet for catalog-level unit testing.

In the module's directory, run unit tests with:

pdk test unit

This command runs all available unit tests.

Module Compatibility

PDK Version Compatibility: Modules created with PDK validate against and run on all Puppet and Ruby version combinations currently under maintenance (see https://docs.puppet.com/puppet/latest/about_agent.html and https://puppet.com/misc/puppet-enterprise-lifecycle)

Contributing

PDK encourages community contributions. See the CONTRIBUTING.md file for development notes.

License

This codebase is licensed under Apache 2.0. However, the open source dependencies included in this codebase might be subject to other software licenses such as AGPL, GPL2.0, and MIT.

Contact Information

• For reporting bugs
• To contact us with questions, join the Puppet Community on Slack

pdk's Issues

pdk executable not installed in path on Debian (8.8 Jessie)

I'm using pdk v1.0.1 on Debian 8.8 Jessie, installed via the official pdk_1.0.1.0-1trusty_amd64.deb package.

When I install the toolkit via sudo dpk -i pdk_1.0.1.0-1trusty_amd64.deb
Then the installation succeeds
But there is no executable pdk found in my path in the shell.

I have to add a symlink to /usr/local/bin to make this work:

sudo ln -sv /opt/puppetlabs/bin/pdk /usr/local/bin/

Allow validation of control repos

The tools included with pdk (puppet-lint, rubocop, etc. as listed in the table on top of the README) are obviously meant to be run via pdk.

Their executables are all located in /opt/puppetlabs/pdk/share/cache/ruby/2.1.0/bin/, not in /opt/puppetlabs/bin/, which does not make them accessible by default from the command line.

There are two shortcomings to this:

1. Some people may like to simply run puppet-lint and other commands directly. The only sane way (from a sysadmin point of view) to fix this is by installing (the Debian package) puppet-lint. This may, however, install a different version of the linter, potentially leading to different validation results.
2. puppet-lint normally also validates a control repository, not only a module. We simply run puppet-lint . in the repository root, usually. Now, pdk validate does not like this idea as you can see below.

$ pdk validate
pdk (FATAL): This command must be run from inside a module (no metadata.json found)

A solution to 1.) would be to make puppet-lint directly accessible by placing it in /opt/puppetlabs/bin/ (or so).

As for 2.) I'm not sure how you guys want us to validate control repositories using the pdk command. I would like to for sure.

PDK `test unit` should cache downloaded modules

When PDK runs unit tests all fixtures (dependencies) are downloaded afresh and deleted completely after completion of the test run.

This is convenient, but when you have a lot of external dependencies (forge_modules or repositories in your .fixtures.yml file) the "Preparing to run the unit tests" step simply takes too long to execute in a test run.

Proposed Solution

PDK's test command should

• either always cache downloaded fixtures, and provide both a --no-cache option to ignore the cache and a pdk clean command to remove downloaded fixtures manually (All test execution frameworks I know do it like this)
• or have a --cache-fixtures option to stop PDK from deleting the downloaded fixtures

Support for future parser on Puppet < 4.0.0

It doesn't seem that there's support for testing against the future parser on older Puppet versions. In the past, I've seen this set with the environment variable FUTURE_PARSER (https://github.com/puppetlabs/puppetlabs_spec_helper/blob/master/lib/puppetlabs_spec_helper/rake_tasks.rb#L509). I'm guessing a CLI option makes more sense with PDK.

Remove EOL style cop from default configuration

Environment:

• PDK Version 1.0.1

Issue:
When generating a module via pdk it creates a .gitattributes with the following contents.

*.rb eol=lf
*.erb eol=lf
*.pp eol=lf
*.sh eol=lf

But when attempting to use the inbuilt validation on a repo thats been pushed through git atleast once
pdk validate ruby -a
it fails complaining the Carriage Return is missing as the files have been made LF instead of CRLF.

PS C:\cygwin64\git\Puppet4Work\pdktest> pdk validate ruby -a
[X] Checking Ruby code style (/.rb)
convention: rubocop: spec/spec_helper.rb:1:1: Carriage return character missing.

Is it possible to modify the validation to prefer LF instead?

Provide official docker image with pdk

As various CI solution, such as GitLab CI, are using Docker containers to run the jobs inside them, it would be nice if you could provide official Docker image with pdk. Preferably it should be based on Alpine for minimal size.

Write .fixtures.yml based on metadata.json

Testing PDK 1.0.0 on MacOS X 10.12.6 creates a module structure that lacks a /.fixtures.yml file.

When running pdk test unit there are no dependency fixtures deployed because of this.

Only the link back to the module in /spec/fixtures/modules/<name of module> is written when pdk test unit is run.

If a /.fixtures.yml file is manually written it will be honored.

The tool could write a /.fixtures.yml file based on /metadata.json or at least deploy with a basic commented-out version that suggests the common tasks.

Common tasks include adding a forge dependency, GitHub or GitLab repository and linking to existing local modules.

File mode of generated files and directories are wrong

I'm using pdk v1.0.1 on Debian 8.8 Jessie, installed via the official pdk_1.0.1.0-1trusty_amd64.deb package.

When I create a new module via the pdk new module <modulename> command
then the permissions of the files are all -rw-rw-rw- (666)
and the permissions of the directories are all drwxrwxrwx+ (777).

Instead, the permissions should be the system/user default, typically 644 (files) and 755 (directories).

Installing Gemfile dependencies on Windows fails

I did a clean install of PDK 1.1.0.0 yesterday, Windows 10, and trying to run pdk validate in a module:

PS C:\Users\dwitt\Documents\git\ahead> pdk validate
pdk (INFO): Running all available validators...
[X] Installing missing Gemfile dependencies.
pdk (FATAL): Fetching https://github.com/bobtfish/hiera-puppet-helper.git

Retrying `git clone --no-checkout --quiet "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-helper-03d5c001b37179971b
a1a942818bb895676603a0" "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper-155f132c0b22"` due to error (2/4): Bundle
r::Source::Git::GitCommandError Git error: command `git clone --no-checkout --quiet "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/
git/hiera-puppet-helper-03d5c001b37179971ba1a942818bb895676603a0" "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper
-155f132c0b22"` in directory C:/Users/dwitt/Documents/git/ahead has failed.
If this error persists you could try removing the cache directory 'C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-h
elper-03d5c001b37179971ba1a942818bb895676603a0'
Retrying `git clone --no-checkout --quiet "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-helper-03d5c001b37179971b
a1a942818bb895676603a0" "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper-155f132c0b22"` due to error (3/4): Bundle
r::Source::Git::GitCommandError Git error: command `git clone --no-checkout --quiet "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/
git/hiera-puppet-helper-03d5c001b37179971ba1a942818bb895676603a0" "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper
-155f132c0b22"` in directory C:/Users/dwitt/Documents/git/ahead has failed.
If this error persists you could try removing the cache directory 'C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-h
elper-03d5c001b37179971ba1a942818bb895676603a0'
Retrying `git clone --no-checkout --quiet "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-helper-03d5c001b37179971b
a1a942818bb895676603a0" "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper-155f132c0b22"` due to error (4/4): Bundle
r::Source::Git::GitCommandError Git error: command `git clone --no-checkout --quiet "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/
git/hiera-puppet-helper-03d5c001b37179971ba1a942818bb895676603a0" "C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper
-155f132c0b22"` in directory C:/Users/dwitt/Documents/git/ahead has failed.
If this error persists you could try removing the cache directory 'C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-h
elper-03d5c001b37179971ba1a942818bb895676603a0'
Git error: command `git clone --no-checkout --quiet
"C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-helper-03d5c001b37179971ba1a942818bb895676603a0"
"C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/bundler/gems/hiera-puppet-helper-155f132c0b22"`
in directory C:/Users/dwitt/Documents/git/ahead has failed.
If this error persists you could try removing the cache directory
'C:/Users/dwitt/AppData/Local/PDK/cache/ruby/2.1.0/cache/bundler/git/hiera-puppet-helper-03d5c001b37179971ba1a942818bb895676603a0'

error: cannot spawn sh: No such file or directory
fatal: unable to fork
error: cannot spawn sh: No such file or directory
fatal: unable to fork
error: cannot spawn sh: No such file or directory
fatal: unable to fork
error: cannot spawn sh: No such file or directory
fatal: unable to fork

pdk (FATAL): Unable to install missing Gemfile dependencies.

Expose other executables into main bin directory

I want to use the git that ships with the pdk but it is nested inside a private directory.

/opt/puppetlabs/pdk/private/git/bin

I also want to use gem, irb, and ruby and any other executables that get shipped with gems

`new module` docs differ from reality

The README has an example of the modulename of the new module command https://github.com/puppetlabs/pdk#module_name

Required. Specifies the name of the module being created, including the namespace. e.g.: username-my_module

When running with this example the run fails:

pdk new module username-my_module
pdk (FATAL): 'username-my_module' is not a valid module name.
Module names must begin with a lowercase letter and can only include lowercase letters, digits, and underscores.

Relates to #157 in that it doesn't appear possible to specify the namespace.

Running PDK native packages on Windows under ConEmu fails

Output from running pdk

PS > Get-Module -Name PuppetDevelopmentKit

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     0.4.0      PuppetDevelopmentKit                pdk


PS C:\Users\garet> pdk
ConEmu blocks ANSICON injection
ConEmu blocks ANSICON injection
ConEmu blocks ANSICON injection
Program 'ansicon.exe' failed to run: The system cannot find the file specifiedAt C:\Program
Files\WindowsPowerShell\Modules\PuppetDevelopmentKit\PuppetDevelopmentKit.psm1:8 char:3
+   &$env:DEVKIT_BASEDIR\private\tools\bin\ansicon.exe $env:RUBY_DIR\bi ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.
At C:\Program Files\WindowsPowerShell\Modules\PuppetDevelopmentKit\PuppetDevelopmentKit.psm1:8 char:3
+   &$env:DEVKIT_BASEDIR\private\tools\bin\ansicon.exe $env:RUBY_DIR\bi ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

More context: cmderdev/cmder#84

Missing bins should not be fatal

pdk (FATAL): Failed to generate binstubs for 'rubocop':
Could not find gem 'rubocop'.

If you don't want to use or can't install a requisite, it should be a warning and not fatally stop the checks

feature request: pdk publish command

It puts a real hamper in my CI/CD processes when I have to manually go upload a tarball to the forge in order to release a new update.

With a native development solution like this, is it finally time to embrace the final step of the process, and include a pdk publish command?

PDK should have an option to disable progress indicator to make it usable in CI

The fancy rotating thing that is shown during pdk validate / test ( [|] [/] [-] ...] ) results in endless outputs when pdk is used in CI pipelines.
An option to disable it (or the whole removal of the fancy buy mostly useless progress indicator) would make pdk a first choice tool in CI.

Wrong URL in module interview

In the license step of the interview the hint is pointing at the wrong URL.

Expected

This should be an identifier from https://spdx.org/licenses/.

Actual

This should be an identifier from https://spdk.org/licenses/.

The domain redirects to a personal domain https://sphericalduck.org/.

PDK validate should have the ability to exclude directories

I am using pdk on my control repo and would like the ability to ignore everything under my modules directory and only test my code.
something like:
'pdk validate --exclude modules/'

add in readline support to ruby

The ruby bundled with pdk seems to not have readline support compiled in.

Post-new module hook

It would be handy to have a way to expose a hook for doing post-create hooks, such as auto-check into git, etc.

PDK RPM for EL7 not relocatable

I noticed when trying to install via RPM on a RHEL 7.3 server as my local account, it bombed on not having access to install, as would be expected. When I tried to --prefix= it told me it was not relocatable.

The gem version had no issues installing in my Dev environment in my local account.

Not a big issue, but something to consider on the RPM builds for developers that don't have Vagrant access and are installing in their home directories.

Default Gemfile for new module need linting

The Gemfile which is created by default on a new module has some style issues that could be fixed.

This is the code quality review from Codacy, on the Gemfile or our control-repo which has been copied from PDK module skeleton:

https://www.codacy.com/app/example42/psick/file/9408172385/issues/source?bid=5179524&fileBranchId=5179524

If you are interested in a PR which addresses those issues, I can prepare one.

Fails to create new task on OSX

Prior to this, I used PDK to create a new module, added PDK to the Gemfile in the new module then ran bundle install from the module directory.

$ bundle exec pdk new task hello_world
bundler: failed to load command: pdk (/Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/bin/pdk)
NameError: uninitialized constant Puppet::Util::Platform
Did you mean?  RUBY_PLATFORM
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/puppet-5.3.2-universal-darwin/lib/puppet/util/windows.rb:15:in `<module:Windows>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/puppet-5.3.2-universal-darwin/lib/puppet/util/windows.rb:1:in `<top (required)>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/util/windows.rb:6:in `require'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/util/windows.rb:6:in `<top (required)>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/util.rb:5:in `require'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/util.rb:5:in `<top (required)>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/cli/exec_group.rb:4:in `require'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/cli/exec_group.rb:4:in `<top (required)>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/cli.rb:8:in `require'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/lib/pdk/cli.rb:8:in `<top (required)>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/exe/pdk:3:in `require'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/gems/pdk-1.2.0/exe/pdk:3:in `<top (required)>'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/bin/pdk:23:in `load'
  /Users/jere/src/bolt/pdk_mod/.bundle/gems/ruby/2.3.0/bin/pdk:23:in `<top (required)>'

New release of tty-spinner

Hey,

I had received some awesome contributions from @austb and wanted to let you know that I've released new version v0.7.0 of tty-spinner. The most important change regards display order of registered spinners read more. Also, you don't have to start top level spinner anymore as it is automatically started when any registered child spinners start spinning. Hope it helps.

Cannot find bundler

Testing 0.4.2 running 'pdk validate' results in:

$ pdk validate -d    
pdk (INFO): Running all available validators...
pdk (DEBUG): PDK package installation not found, trying 'bundle' from the system PATH instead
[-] Checking for missing Gemfile dependenciespdk (DEBUG): Executing 'bundle check --gemfile=/home/dwittenberg/github/modules/ahead/Gemfile --path=/home/dwittenberg/.pdk/cache'
[���] Checking for missing Gemfile dependencies

/opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/dependency.rb:298:in `to_specs': Could not find 'bundler' (>= 0) among 8 total gem(s) (Gem::LoadError)
	from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/dependency.rb:309:in `to_spec'
	from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:53:in `gem'
	from /opt/puppetlabs/puppet/bin/bundle:22:in `<main>'
pdk (DEBUG): PDK package installation not found, trying 'bundle' from the system PATH instead
[-] Installing missing Gemfile dependenciespdk (DEBUG): Executing 'bundle install --gemfile=/home/dwittenberg/github/modules/ahead/Gemfile --path=/home/dwittenberg/.pdk/cache'
[���] Installing missing Gemfile dependencies

/opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/dependency.rb:298:in `to_specs': Could not find 'bundler' (>= 0) among 8 total gem(s) (Gem::LoadError)
	from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/dependency.rb:309:in `to_spec'
	from /opt/puppetlabs/puppet/lib/ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:53:in `gem'
	from /opt/puppetlabs/puppet/bin/bundle:22:in `<main>'
pdk (FATAL): Unable to install missing Gemfile dependencies.
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/pdk-0.4.2/lib/pdk/util/bundler.rb:32:in `ensure_bundle!'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/pdk-0.4.2/lib/pdk/cli/validate.rb:77:in `block (2 levels) in <module:CLI>'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/cri-2.9.1/lib/cri/command.rb:319:in `call'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/cri-2.9.1/lib/cri/command.rb:319:in `run_this'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/cri-2.9.1/lib/cri/command.rb:264:in `run'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/cri-2.9.1/lib/cri/command.rb:282:in `run'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/pdk-0.4.2/lib/pdk/cli.rb:15:in `run'
pdk (DEBUG): /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/pdk-0.4.2/exe/pdk:5:in `<top (required)>'
pdk (DEBUG): /opt/puppetlabs/puppet/bin/pdk:23:in `load'
pdk (DEBUG): /opt/puppetlabs/puppet/bin/pdk:23:in `<main>'

$ which gem
/opt/puppetlabs/puppet/bin/gem

$ gem list |grep -i bundler
bundler (1.15.2, 1.15.1, 1.13.7)

How to run beaker with pdk?

I've been playing with pdk, and wanted to test it in a clean environment so I've spun up an Ubuntu Trusty virtual machine with Vagrant, and installed Ruby 2.3 from the Brightbox PPA, before installing pdk from Rubygems.

The generated module includes a Rakefile, and that Rakefile includes targets for running Beaker:

vagrant@vagrant-ubuntu-trusty-64:~/example$ bundle exec rake -T
rake beaker                    # Run beaker acceptance tests
rake beaker:default            # Run the Beaker acceptance tests for the node set 'default'
rake beaker:sets               # List available beaker nodesets
rake beaker:ssh[set,node]      # Try to use vagrant to login to the Beaker node
rake beaker:ssh:default[node]  # Use vagrant to login to a node from the set 'default'
rake build                     # Build puppet module package
rake check:dot_underscore      # Fails if any ._ files are present in directory
rake check:git_ignore          # Fails if directories contain the files specified in .gitignore
rake check:symlinks            # Fails if symlinks are present in directory
rake check:test_file           # Fails if .pp files present in tests folder
rake clean                     # Clean a built module package
rake compute_dev_version       # Print development version of module
rake help                      # Display the list of available rake tasks
rake lint                      # Run puppet-lint
rake metadata_lint             # Run metadata-json-lint
rake parallel_spec             # Parallel spec tests
rake release_checks            # Runs all necessary checks on a module in preparation for a release
rake rubocop                   # Run RuboCop
rake rubocop:auto_correct      # Auto-correct RuboCop offenses
rake spec                      # Run spec tests and clean the fixtures directory if successful
rake spec_clean                # Clean up the fixtures directory
rake spec_prep                 # Create the fixtures directory
rake spec_standalone           # Run spec tests on an existing fixtures directory
rake syntax                    # Syntax check Puppet manifests and templates
rake syntax:hiera              # Syntax check Hiera config files
rake syntax:manifests          # Syntax check Puppet manifests
rake syntax:templates          # Syntax check Puppet templates
rake validate                  # Check syntax of Ruby files and call :syntax and :metadata_lint

and I have a single node defined in the nodeset

vagrant@vagrant-ubuntu-trusty-64:~/example$ bundle exec rake beaker:sets
default

however bundle exec rake beaker:default does nothing, and it seems that Beaker itself is not installed?

vagrant@vagrant-ubuntu-trusty-64:~/example$ bundle exec rake beaker:default
/usr/bin/ruby2.3 -I/home/vagrant/.pdk/cache/bundler/ruby/2.3.0/gems/rspec-core-3.6.0/lib:/home/vagrant/.pdk/cache/bundler/ruby/2.3.0/gems/rspec-support-3.6.0/lib /home/vagrant/.pdk/cache/bundler/ruby/2.3.0/gems/rspec-core-3.6.0/exe/rspec spec/acceptance --color
No examples found.

Finished in 0.00046 seconds (files took 0.05271 seconds to load)
0 examples, 0 failures

vagrant@vagrant-ubuntu-trusty-64:~/example$ gem list | grep beaker
vagrant@vagrant-ubuntu-trusty-64:~/example$ bundle exec gem list | grep beaker
vagrant@vagrant-ubuntu-trusty-64:~/example$

am I completely completely misunderstanding something here or should the beaker:default not attempt to create the default instance and then run the tests (and then should it not fail due to there being no Beaker Gem installed?)

Failed to create new module with "No such file or directory - git"

I've been playing with pdk, and wanted to test it in a clean environment so I've spun up an Ubuntu Trusty virtual machine with Vagrant, and installed Ruby 2.3 from the Brightbox PPA, before installing pdk from Rubygems.

On the first try of running pdk new module I get the fatal error:

About to generate this module; continue? Yes
pdk (FATAL): Failed to execute 'git --git-dir /var/lib/gems/2.3.0/gems/pdk-0.3.0/.git describe --all --long': No such file or directory - git
vagrant@vagrant-ubuntu-trusty-64:~$ ls /var/lib/gems/2.3.0/gems/pdk-0.3.0/.git
ls: cannot access /var/lib/gems/2.3.0/gems/pdk-0.3.0/.git: No such file or directory

System information:

vagrant@vagrant-ubuntu-trusty-64:~$ uname -a
Linux vagrant-ubuntu-trusty-64 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
vagrant@vagrant-ubuntu-trusty-64:~$ ruby --version
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux-gnu]
vagrant@vagrant-ubuntu-trusty-64:~$ gem list

*** LOCAL GEMS ***

bigdecimal (1.2.8)
bundler (1.15.1)
childprocess (0.6.3)
colored (1.2)
cri (2.9.1)
deep_merge (1.1.1)
equatable (0.5.0)
fast_gettext (1.1.0)
ffi (1.9.18)
gettext (3.2.3)
gettext-setup (0.25)
io-console (0.4.5)
json (1.8.3)
locale (2.1.2)
necromancer (0.4.0)
pastel (0.7.1)
pdk (0.3.0)
psych (2.1.0)
rdoc (4.2.1)
text (1.3.1)
tty-color (0.4.2)
tty-cursor (0.4.0)
tty-prompt (0.12.0)
tty-spinner (0.4.1)
wisper (1.6.1)

Add `new control` command for creating control repository

PDK should support creating and extending a control-repository, which is enabled for validation and testing using PDK (just like the modules created by PDK).

Obsoletes #289 and #332 when implemented.
See also: puppetlabs/control-repo#52

Proposed Solution

Ideally,

• the validation and test setup currently present in generated modules should be taken over to the proposed Puppet Labs control-repo, and
• the pdk new control command should generate a control-repo setup that is equivalent to the proposed Puppet Labs control-repo on GitHub.

This way, whether you use PDK or just the GitHub repo as an example, you can trust to get a usable best-practice setup.

Running PDK behind a corporate proxy fails

Environment:

• NTLM Auth based Proxy
• Windows 7
• Powershell v5
• PDK v0.5.0.0 (Installed using provided .msi package)

Currently PDK fails to function behind a corporate proxy.

I could find no documentation on this issue, but after some experimenting I found setting the following environment variables in Powershell before running PDK commands allows it to communicate.

$env:http_proxy="http://user:[email protected]:port" $env:HTTP_PROXY="http://user:[email protected]:port" $env:https_proxy="http://user:[email protected]:port" $env:HTTPS_PROXY="http://user:[email protected]:port"

AIX missing when creating new module during Question 5

Currently when creating a new module PDK it only shows the following when going to Question 5:

[Q 5/9] What operating systems does this module support?
Use the up and down keys to move between the choices, space to select and enter to continue.
--> RedHat based Linux, Debian based Linux, Windows (Use arrow or number (1-7) keys, press Space to select and Enter to finish)
  ⬢ 1) RedHat based Linux
  ⬢ 2) Debian based Linux
  ⬡ 3) Fedora
  ⬡ 4) OSX
  ⬡ 5) SLES
  ⬡ 6) Solaris
‣ ⬢ 7) Windows

As a User I would expect that it also asks for AIX during this Question.

Add /bin/ to .gitignore

When following the steps in the readme, after generating the module I ran a git init && git add . && git commit -m "Initial commit". I then ran a pdk validate from the module directory, and noticed that it created a /bin/ directory. Should that be added to the generated .gitignore file?

PDK should have yum/apt/choco repos

On the lines of #323 it would be welcomed a YUM / APT (and eventually choco/whatever fits in macos) repositories to install the pdk package.
The current "download from a redirect link" approach make the unattended installation and upgrade of pdk via puppet classes a bit cumbersome.
Having package repos would benefit upgrades, and increase the number of people using latest versions.

pdk expects missing git binaries

Currently the pdk expects git to exist at /opt/puppetlabs/sdk/private/git/bin/git. However, we install the pdk via gem install pdk which of course does not install git or create anything inside /opt/puppetlabs.

Users can just symlink git to that location but if there's a "correct" way to handle that, I'd love to document it.

Maybe a reasonable solution might be to allow it to search the normal $PATH for git when installed as a Gem instead of as part of the official pdk packaging (which doesn't yet exist).

PDK 1.2.1 `test unit` fails for unsupported OSes

I'm testing roles in our control repository, and I've notices that while pdk unit test passed successfully last week (with PDK 1.2.0) it fails since the release of PDK 1.2.1, e.g.

$ pdk test unit 2> /dev/null
failed: rspec: ./spec/classes/role_server_customer_standalone_spec.rb:10: error during compilation: Evaluation Error: Unknown variable: '::lsbdistcodenameid'. at /builds/comsolit/puppet-control/spec/fixtures/modules/php/manifests/params.pp:38:27 on node runner-4c61c68a-project-186-concurrent-0.comsolit.local
  role::server::customer_standalone on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_backoffice_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-4c61c68a-project-186-concurrent-0.comsolit.local
  role::workstation::backoffice on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

failed: rspec: ./spec/classes/role_workstation_developer_spec.rb:14: error during compilation: Evaluation Error: Error while evaluating a Function Call, The software module is not supported on Debian with version 9.1. at /builds/comsolit/puppet-control/spec/fixtures/modules/software/manifests/params.pp:192:5 on node runner-4c61c68a-project-186-concurrent-0.comsolit.local
  role::workstation::developer on debian-9-x86_64 should compile into a catalogue without dependency cycles
  Failure/Error:
        end
  
        it { is_expected.to compile.with_all_deps }
      end
    end

ERROR: Job failed: exit code 1

Our metadata.json specifies Debian 9 as our target OS version, that is why compiling mayflower-php (or puppet-php) seems to fail:

  "operatingsystem_support": [
    {
      "operatingsystem": "Debian",
      "operatingsystemrelease": [
        "9"
      ]
    },

It is probably intended to more strongly enforce the manifest now, correct?

Function: pdk new module my_module does not work when a user name contains non-alphanumeric characters

My user name contains a dot (it is: davin.hanlon). When I try to create a new module using the 'pdk new module' command I get an error, as below. This is due to the presence of the dot in my username.

davin:~ davin.hanlon$ pdk new module harrumph
pdk (INFO): Creating new module: harrumph
/Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/module/metadata.rb:101:in validate_name': Invalid 'name' field in metadata.json: the namespace contains non-alphanumeric characters (ArgumentError) from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/module/metadata.rb:77:in process_name'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/module/metadata.rb:64:in update!' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/module/metadata.rb:41:in initialize'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/generators/module.rb:35:in new' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/generators/module.rb:35:in invoke'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/cli/new/module.rb:41:in block (2 levels) in <module:CLI>' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/cri-2.9.1/lib/cri/command.rb:319:in run_this'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/cri-2.9.1/lib/cri/command.rb:264:in run' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/cri-2.9.1/lib/cri/command.rb:282:in run'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/cri-2.9.1/lib/cri/command.rb:282:in run' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/lib/pdk/cli.rb:14:in run'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/gems/pdk-0.2.0/exe/pdk:5:in <top (required)>' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/bin/pdk:23:in load'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/bin/pdk:23:in <main>' from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in eval'
from /Users/davin.hanlon/.rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `

'

create_process error on windows when creating a new module

When trying to create a new module using the PDK on windows i recieve the following error

Can't create module if new module fails to get login

I managed to trigger the following exception when running pdk new module, in my case when running under root on Debian.

pdk (INFO): Creating new module: something
/usr/local/bundle/gems/pdk-0.4.1/lib/pdk/generators/module.rb:58:in `username_from_login': undefined method `gsub' for nil:NilClass (NoMethodError)
        from /usr/local/bundle/gems/pdk-0.4.1/lib/pdk/generators/module.rb:71:in `prepare_metadata'
        from /usr/local/bundle/gems/pdk-0.4.1/lib/pdk/generators/module.rb:27:in `invoke'
        from /usr/local/bundle/gems/pdk-0.4.1/lib/pdk/cli/new/module.rb:41:in `block (2 levels) in <module:CLI>'
        from /usr/local/bundle/gems/cri-2.9.1/lib/cri/command.rb:319:in `run_this'
        from /usr/local/bundle/gems/cri-2.9.1/lib/cri/command.rb:264:in `run'
        from /usr/local/bundle/gems/cri-2.9.1/lib/cri/command.rb:282:in `run'
        from /usr/local/bundle/gems/cri-2.9.1/lib/cri/command.rb:282:in `run'
        from /usr/local/bundle/gems/pdk-0.4.1/lib/pdk/cli.rb:15:in `run'
        from /usr/local/bundle/gems/pdk-0.4.1/exe/pdk:5:in `<top (required)>'
        from /usr/local/bundle/bin/pdk:22:in `load'
        from /usr/local/bundle/bin/pdk:22:in `<main>'

Note that getlogin:

Returns the short user name of the currently logged in user.
Unfortunately, it is often rather easy to fool ::getlogin.

From https://github.com/ruby/ruby/blob/202bbda2bf5f25343e286099140fb9282880ecba/ext/etc/etc.c#L60

Reading the source, it suggests falling back to getpwuid. On the same host as above:

> require 'etc'
=> true
> Etc.getlogin
=> nil
> Etc.getpwuid
=> #<struct Etc::Passwd name="root", passwd="x", uid=0, gid=0, gecos="root", dir="/root", shell="/bin/bash">
> Etc.getpwuid.name
=> "root"

Login ends up as nil here: https://github.com/puppetlabs/pdk/blob/master/lib/pdk/generators/module.rb#L57

As a side note, I didn't see a way of overriding this value? For instance if releasing modules under multiple namespaces, or working on computers where usernames are corporate controlled. I didn't see anything in --help. Happy to open a separate issue on that if useful.

Don't buffer output from bundle commands

When I run bundle exec rake spec, I see output immediately. When I run pdk bundle exec rake spec, I have to wait until the end to see output. We have a zillion tests. Is there some way to disable buffering STDIN -> STDOUT (or whatever...)?

Create test layout for control repositories

I want to make sure our control repository is covered by tests.

Currently, I need to figure out myself how to create a test setup. An example suggested by @alvagante in issue #289 is example42/psick.

It would be good if there was a PDK command to create such a test setup, and a command to run the test suite (and validation) of the control repository.

Feature Request - Questionnaire option to select CI tool

An additional question would be nice that asks about which CI tool a user is using and providing a template for that.

Popular CI tools suggested for templates:

• Gitlab CI (.gitlab-ci.yml)
• Jenkins (Jenkinsfile)
• Circle CI
• Appveyor (already included)
• Travis (Already Included)

Validate fails on existing module

With PDK installed via gems into the system ruby (2.4.0 in this case). Running validate in a module (puppetlabs-apache here) throws the following error.

pdk (INFO): Running all available validators...
[✖] Resolving Gemfile dependencies

/usr/local/lib/ruby/site_ruby/2.4.0/rubygems.rb:270:in `find_spec_for_exe': can't find gem bundler (>= 0.a) (Gem::GemNotFoundException)
        from /usr/local/lib/ruby/site_ruby/2.4.0/rubygems.rb:298:in `activate_bin_path'
        from /usr/local/bundle/bin/bundle:22:in `<main>'
pdk (FATAL): Unable to resolve Gemfile dependencies.

Note pdk and bundler are both installed:

gem list

*** LOCAL GEMS ***

bigdecimal (default: 1.3.0)
bundler (1.15.1, 1.14.6)
childprocess (0.6.3)
colored (1.2)
cri (2.9.1)
deep_merge (1.1.1)
did_you_mean (1.1.0)
equatable (0.5.0)
fast_gettext (1.1.0)
ffi (1.9.18)
gettext (3.2.3)
gettext-setup (0.26)
io-console (default: 0.4.6)
json (default: 2.0.2)
json_pure (2.1.0)
locale (2.1.2)
minitest (5.10.1)
necromancer (0.4.0)
net-telnet (0.1.1)
openssl (default: 2.0.2)
pastel (0.7.1)
pdk (0.4.1)
power_assert (0.4.1)
psych (default: 2.2.2)
rake (12.0.0)
rdoc (default: 5.0.0)
rubygems-update (2.6.11)
test-unit (3.2.3)
text (1.3.1)
tty-color (0.4.2)
tty-cursor (0.4.0)
tty-prompt (0.12.0)
tty-spinner (0.4.1)
wisper (1.6.1)
xmlrpc (0.2.1)

and bundle is on the path:

which bundle
/usr/local/bundle/bin/bundle

Internal Server Error on PDK Download site

Not sure if you guys have already noticed, but the download site seems to be broken.

All the links to PDK packages (installers) of the various OS flavors lead to an Error 500 page.

Repo Configs Contain Invalid URLs

The configs for the pdk repos, hosted in s3, are generated with invalid urls.

[karmix@localhost ~]$ curl https://puppet-pdk.s3.amazonaws.com/pdk-latest/repo_configs/rpm/pl-pdk-latest-el-7-x86_64.repo
[pl-pdk-latest]
name=PL Repo for pdk at commit latest
baseurl=http://https://s3.amazonaws.com/puppet-pdk-latest//pdk-latest/repos/el/7/PC1/x86_64/
enabled=1
gpgcheck=1
gpgkey=http://https://s3.amazonaws.com/puppet-pdk-latest//7F438280EF8D349F
[karmix@localhost ~]$ 

Note that:

1. The URLs for baseurl, and gpgkey appear to have the s3 bucket URL "https://s3.amazonaws.com/puppet-pdk-latest/" embedded in them where you would expect to find a hostname.
2. The bucket puppet-pdk-latest does not actually exist (the bucket name is puppet-pdk).
3. The GPG key does not appear to be present at the corrected URL, or anywhere in the bucket.
4. All of the other repo configs, for both deb and rpm, are generated with the same URL problems.

I would expect the config to look something more like:

[karmix@localhost ~]$ cat /etc/yum.repos.d/pl-pdk-latest-el-7-x86_64.repo 
[pl-pdk-latest]
name=PL Repo for pdk at commit latest
baseurl=http://s3.amazonaws.com/puppet-pdk/pdk-latest/repos/el/7/PC1/x86_64/
enabled=1
gpgcheck=1
gpgkey=https://yum.puppetlabs.com/RPM-GPG-KEY-puppet
[karmix@localhost ~]$ 

pdk validate fails if host puppet.conf contains deprecated settings

Causes jenkins jobs to go unstable as this is logged as test failures when published as junit test results.

This should not happen as the contents of the module under test shouldn't have anything to do with the host configuration. It would make more sense for pdk to contain a stub puppet.conf & override reference to that when puppet is invoked.

Repro scenario:
host happens to be managed by an older version of puppet.
The system puppet.conf on the host contains 'configtimeout'
command to run: pdk validate --format=junit:validate.xml && exit %ERRORLEVEL%

output:

...
<testsuite name="puppet-syntax" tests="3" errors="0" failures="2" skipped="0" time="0" timestamp="2017-09-29T15:49:37" hostname="hosty" id="2" package="puppet-syntax">
<properties/>
<testcase classname="puppet-syntax" name="manifests/init.pp" time="0"/>
<testcase classname="puppet-syntax" name="" time="0">
<failure type="Warning" message="Setting configtimeout is deprecated.">
Warning: puppet-syntax: Setting configtimeout is deprecated.
</failure>
</testcase>
<testcase classname="puppet-syntax" name="" time="0">
<failure type="" message="(at C:/Program Files/Puppet Labs/DevelopmentKit/share/cache/ruby/2.1.0/gems/puppet-5.2.0-x64-mingw32/lib/puppet/settings.rb:1159:in `issue_deprecation_warning')">
: puppet-syntax: (at C:/Program Files/Puppet Labs/DevelopmentKit/share/cache/ruby/2.1.0/gems/puppet-5.2.0-x64-mingw32/lib/puppet/settings.rb:1159:in `issue_deprecation_warning')
</failure>
</testcase>
<system-out/>
<system-err/>
</testsuite>
...

PDK should be able to modify modules setup, add features, etc.

Currently, the PDK is designed to generate a module skeleton. IIUC, it is not possible to adjust the module setup (e.g. add supported OS, change software license, update URLs and Puppet version requirement, replace a CI configuration).

I know this may sound stupid or exaggerated (because "you can simply edit the metadata file"), but it should be possible to have PDK give us a hand not only for generating an initial setup but also modify the very same parameters later.

Non-trivial UX Feature

This request must be seen from a development speed and error avoidance point of view ("... simply do the same thing again for updating").

Please let me acknowledge that this may not be a trivial task to accomplish.

Include erb/epp validators

Since templates are pretty important erb and epp should both include validators

Support for fedora

Any plans in building RPM's for fedora ?

Gems not found in pre-release

$ pdk --version
1.0.0.pre

$ pdk validate
pdk (DEBUG): PDK package installation not found, trying 'bundle' from the system PATH instead
pdk (DEBUG): Executing 'bundle binstubs rubocop --force'
pdk (DEBUG): Execution of 'bundle binstubs rubocop --force' complete (duration: 0.39786395s; exit code: 7)
pdk (FATAL): Failed to generate binstubs for 'rubocop':
Could not find gem 'rubocop'.

$ gem list rubocop

*** LOCAL GEMS ***

rubocop (0.49.1)

Include yaml validator

Since yaml files are used a lot in hiera there should be a validator for .yaml files

Request for portable Windows version

There should be a version available for Windows that can simply be unpacked, or otherwise installed without needing administrative rights.

configuration of file include/exclude patterns for validation

(see the rest of #329 for related discussion)

PDK currently uses relatively permissive patterns to build a list of files to scan for validation and supplies them as arguments to the various validators when they are invoked. This works in many cases, but is problematic when a module contains a file with an unexpected extension or when (for better or worse) external code is vendored into a module.

It is possible to configure a specific include pattern when performing pdk validate, but this include is shared among all the validators which throw errors when they cannot match their expected file types. See #329 (comment) for an example. This also appeared to trigger an early exit with return code 0 when I tried it.

Without PDK, Puppet testing is typically done with a Rakefile that contains individual configuration for some of the file patterns for puppet-lint and puppet-syntax gems while a .rubocop.yml file is often used to set exclusions for Ruby code. This is a bit complicated, but allows working around any issues of this nature. PDK simplifies the configuration by calling the validation tools directly, but removes the ability to configure the patterns.

PDK presents an interesting opportunity to centralize and simplify the include/exclude pattern logic, possibly with a PDK specific configuration file. Is this or any alternatives on the table for a future PDK release?