voxpupuli / puppet-download_file

Puppet define created that can be used to download files rather than needing to store binaries in puppet repos

Download File module for Puppet

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with download_file
   • What download_file affects
   • Setup requirements
   • Beginning with download_file
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module

Overview

The download_file module allows you to download files on Windows

Module Description

The download_file module introduced a small define download_file that will allow you to download a file over http(s) for usage during an installation. This was created because the package resource does not support http as a source for packages. It is only supported on Windows.

Setup

What download_file affects

• Downloads files onto each node

Setup Requirements

• download_file makes use of Powershell so you will need to have at least version 2.0 installed in order to use this module.

Beginning with download_file

To download dotnet 4.0

    download_file { "Download dotnet 4.0" :
      url                   => 'http://download.microsoft.com/download/9/5/A/95A9616B-7A37-4AF6-BC36-D6EA96C8DAAE/dotNetFx40_Full_x86_x64.exe',
      destination_directory => 'c:\temp'
    }

To download dotnet 4.0 using a proxy

    download_file { "Download dotnet 4.0" :
      url                   => 'http://download.microsoft.com/download/9/5/A/95A9616B-7A37-4AF6-BC36-D6EA96C8DAAE/dotNetFx40_Full_x86_x64.exe',
      destination_directory => 'c:\temp',
      proxy_address         => 'http://corporateproxy.net:8080'
    }

The proxy will be used as part of the download using PowerShell. This does not set a system wide proxy

Usage

Reference

Defined Types

Public Types

• download_file: Download a give file

Parameters

url

The http(s) destination of the file that you are looking to download

destination_directory

The full path to the directory on the system where the file will be downloaded to

destination_file

The optional name of the file to download onto the system.

user

The optional userid is used when authenticating via Basic Auth with the server. user and password must be specified or this has no effect.

password

The optional password is used when authenticating via Basic Auth with the server. user and password must be specified or this has no effect.

proxy_address

The optional http proxy address to use when downloading the file

proxy_user

The optional http proxy user to use when downloading the file. proxy_address and proxy_password must be specified or this has no effect.

proxy_password

The optional http proxy password to use when downloading the file. proxy_address and proxy_user must be specified or this has no effect. By default this value accepts secure strings. A secure string is (unfortunately) tied to the machine that it is used for. To generate a secure string for a given machine, users should run the following powershell command on that machine (replacing PASSWORD with the desired password):

ConvertFrom-SecureString -securestring $(ConvertTo-SecureString "PASSWORD" -AsPlainText -Force)

It is possible to get this information then clear the command from history, but it's important to note that the -Force argument is there to suppress warnings that the plaintext password is in the history.

If this process sounds unappealing, you can send the password in plaintext (which sits in the download-<filename>.ps1 file on the machine being provisioned) by changing the is_password_secure variable to false.

is_password_secure

The optional switch to change the way that proxyPassword is interpreted from secure string to plaintext. This will send the password in plaintext to the machine being provisioned, which may be a security concern.

cookies

An optional array of cookies to add to the HTTP request for the download.

Limitations

This module is tested on the following platforms:

• Windows 2008
• Windows 2008 R2
• Windows 2012
• Windows 2012 R2
• Windows 7
• Windows 8

It is tested with the OSS version of Puppet only.

Contributing

Please read CONTRIBUTING.md for full details on contributing to this project.

puppet-download_file's Issues

Download from github.com returns WebException The request was aborted: Could not create SSL/TLS secure channel.

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: Puppet 5.5.6 Client / Puppetserver 2.8.1 (from pc1-1.1.0)
• Module version: 3.1.0
• OS: Client running on Windows Server 2016

How to reproduce (e.g Puppet code you use)

download_file { $installer:
  url                   => 'https://github.com/git-for-windows/git/releases/download/v2.19.1.windows.1/Git-2.19.1-64-bit.exe',
  destination_directory => 'c:\\'
}

What are you seeing

When I run Puppet agent manually I see the following error:

Notice: /Stage[main]/Git::Install::Windows/Download_file[Git-2.19.1-64-bit.exe]/File[download-Git-2.19.1-64-bit.exe.ps1]/ensure: defined content as '{md5}4a913e51b88e77f9750f702cdc6c02c7'
Notice: /Stage[main]/Git::Install::Windows/Download_file[Git-2.19.1-64-bit.exe]/Exec[download-Git-2.19.1-64-bit.exe]/returns: System.Net.WebException
Notice: /Stage[main]/Git::Install::Windows/Download_file[Git-2.19.1-64-bit.exe]/Exec[download-Git-2.19.1-64-bit.exe]/returns: The request was aborted: Could not create SSL/TLS secure channel.
Notice: /Stage[main]/Git::Install::Windows/Download_file[Git-2.19.1-64-bit.exe]/Exec[download-Git-2.19.1-64-bit.exe]/returns:
Error: 'c:\\download-Git-2.19.1-64-bit.exe.ps1' returned 1 instead of one of [0]
Error: /Stage[main]/Git::Install::Windows/Download_file[Git-2.19.1-64-bit.exe]/Exec[download-Git-2.19.1-64-bit.exe]/returns: change from 'notrun' to ['0'] failed: 'c:\\download-Git-2.19.1-64-bit.exe.ps1' returned 1 instead of one of [0]

What behaviour did you expect instead

I expected the Git executable to downloaded to C:\Git-2.19.1-64-bit.exe

Output log

See above.

Any additional information you'd like to impart

A potential fix is to add the following into the try block in the download PowerShell script:

[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11"

For example, in the case of the PowerShell script generated by the above Puppet code:

try {
  [Net.ServicePointManager]::SecurityProtocol = "tls12, tls11"
  $webclient.DownloadFile('https://github.com/git-for-windows/git/releases/download/v2.19.1.windows.1/Git-2.19.1-64-bit.exe', 'c:\\Git-2.19.1-64-bit.exe')
}

When I run the PowerShell script manually in ISA the download completes successfully.

There may be issues around setting the security protocol like this both outside of Windows Server 2016 and for remote URLs that only support tls1.0 or ssl3 (which would be Very Bad but folk might expect download_file to still work).

Name the PowerShell script after destination file if set

If the $destination_file variable is set then the PowerShell script should use that variable instead of getting the filename from the URL.

In my case, I'm downloading two files with the same name in the URL (two different files that happens to have the same name in the URI) which causes the content of this file be constantly rewritten. Since there is a dependency between that file and the downloading, the files get redownloaded each time the agent runs.

Be able to specify destination file-name

It would be helpful to be able to specify the "@filename" variable when calling download_file. I am trying to download files for installers that have a URL that ends in "?raw" like notepad++.msi?raw . I am going to have to fork your code to make it work for me I think but would rather avoid that.

No authentication

I need to download new build from build server which requires authentication. At the moment we can't do that with this module.

Temp resource created without being used

In: https://github.com/puppet-community/puppet-download_file/blob/master/manifests/init.pp#L64-L80

The Temp resource is created without ever being used, it's a require for the file on line 80, but isn't actually _used_ in the download directory and so is pointless to ensure this exists.

Removing this code would make more sense and also conflicts with other definitions of C:\Temp created with ensure_resource for example.

Error when the filename in the URL has spaces

When I have a resource like this:

download_file { 'Download "derp I have spaces.txt"':
  destination_directory => 'C:/Users/Administrator/Desktop',
  url                   => 'https://fileserver/derp I have spaces.txt',
}

The regex here, https://github.com/voxpupuli/puppet-download_file/blob/master/manifests/init.pp#L68, only captures the first word in the filename, resulting in an Exec resource like this:

exec { "download-derp I have spaces.txt":
  command   => "C:/Users/Administrator/Desktop\\download-derp.ps1",
  provider  => powershell,
  onlyif    => "if(Test-Path -Path '"C:/Users/Administrator/Desktop\\derp I have spaces.txt') { exit 1 } else { exit 0 }",
  logoutput => true,
  require   => File["download-derp I have spaces.txt.ps1"],
}

Because of the spaces, the command loses the whole filename, which results in Powershell erring with The term 'C:/Users/Administrator/Desktop\\download-derp.ps1' is not recognized as the name of a cmdlet, function, script file, or operable program...

For now, I'll try working around this by encoding the URL. If that fails, then I'll rename the file on our fileserver.

Use Windows temp dir

In PR #23, @noma4i commented:

Windows default TEMP folder now is %userprofile%\AppData\Local\Temp c:\temp may not exist at fresh windows instance. We use $env:TEMP in powershell templates to get proper path.

The generated Powershell scripts should probably do just that? Can you comment on the compatibility with older releases of Windows and Powershell?

puppetlabs/stdlib version below 6.0.0

Hello,

Is there any particular reason why puppetlabs/stdlib version is bound below 6.0.0? The latest version is 6.2.0 and I'm experiencing dependency conflicts when mixing download_file with other modules.

Regards,

xtimmu

Use ruby api to download file instead of powershell.

Wouldn't it be simplier to use the Ruby built-in Net::HTTP api for the file transfer instead of using PowerShell?

Delete powershell file after download complete

Please add an option to delete the powershell file used to download the requested file after the download has completed.

SSL Error

If you have a proxy that has some issues when downloading files from SSL you may need to import the SSL Certificates or you can end up having the error:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

To avoid that, you can add this line at the beginning of the download.ps1.erb template:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

That happened to me and this line fixed this issue.

Could you please add this line in the template to avoid this error in the future?

Webclient doesn't support downloading URLs with query-strings

I came across this today when trying to use this module to download a git repo's zip file.

I think to support this, you could use a different web client or take in a new param to hold the query-string (and add that object to the webclient request).

Download works but throws an error

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 4.10 puppet server with foreman version 1.12 and puppet agent 1.10.10
• Distribution: linux server windows client
• Module version: latest

How to reproduce (e.g Puppet code you use)

Something like this
download_file { 'download' :
url => 'http://webpage.com/file.zip',
destination_directory => 'c:\tmp',
}

What are you seeing

The return of the command is 1 instead of 0 but it is successfully downloaded

What behaviour did you expect instead

Not getting an error

Any additional information you'd like to impart

Its a 1,2GB zip maybe its becaus it tooks a lot of time to download? Aprox 5-10 min

puppetlabs-powershell got updated in january to 3.0.1

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet:
• Ruby:
• Distribution:
• Module version: 4.0.0

Any additional information you'd like to impart

Can't the dependencies be updated to reflect the newest release or is was there really an breaking change introduced with the newer powershell module version?
Link to the module: https://forge.puppet.com/puppetlabs/powershell

Windows Nano Server support

This module does not work on Nano Server due .Net's "System.Net.WebClient" not being available in the CoreCLR.

"System.Net.Http.HttpClient" is a viable alternative as used for example here:
https://github.com/cloudbase/unattended-setup-scripts/blob/master/FastWebRequest.psm1

Command timeout for big files

Hi,
I think there should be a parameter for exec timeout, cause default timeout for exec command is 300s. When you have a slower internet connection or when you try to download a bigger file then command failes. What is your opinion about it ? thanks

Download to windows Folder

I am tying to download background to default windows location and seems that are some issues

download_file { "Windows-backgroundDefault" :
url => 'http://puppet/puppetfiles/BackgroundDefault.jpg',
destination_directory => 'C:/Windows/System32/oobe/INFO/BACKGROUNDS'
}

Anyone tried to copy anything to Windows folder?

Dependency on c:\temp causes collision/failure to apply catalog

Beginning at line 10 in the init.pp for the shared download_file has:

if ! defined(File['C:\temp']) {
    file { 'C:\temp':
      ensure => directory,
    }
  }

I specified a resource in one of my classes as follows

$temp_dir = 'C:/TEMP'
$build_dir = D:/BUILD'
file{ [$temp_dir, $build_dir]:
      ensure   => 'directory',
}

I received the following (some folders changed to obscure specific info)

Failed to apply catalog: Cannot alias File[C:\temp] to ["C:/TEMP"] at /app/SOMEDIR/puppet/conf/environments/shared/download_file/manifests/init.pp:12; resource ["File", "C:/TEMP"] already declared at /app/SOMEDIR/puppet/conf/environments/myenvironment_dev/modules/my_apptier/manifests/create_folders.pp:26

Conclusion

It appears that the check to see if c:\temp is defined in download_file does not take into account either the case or the backslash vs forward slash - not sure which one.

Work-Around

To work around for now, I have ensured that I declare as follows so that it matches the exact resource and the "if ! defined" works.

$temp_dir = 'C:\temp'
$build_dir = D:/BUILD'
file{ [$temp_dir, $build_dir]:
      ensure   => 'directory',
}

Suggestion

Can the check if it's already defined account for case variations or forward/backward slash variations?

Fails to download when the destination_directory has space character

We are trying to download a file under a directory containing space character and it is failing even after escaping it with (` or ).

Resource:
download_file {'file download': {
url => 'http://test.com/file.ps1',
destination_directory => 'C:\Program Files\Test',
}

Error:
Notice: /Stage[main]/Test_module/Download_file[file download]/Exec[download-file.ps1]/returns: C:/Program : The term 'C:/Program' is not recognized as the name of a cmdlet, function, script file, or operable

Workaround:
Updating line number 102 in download_file/manifests/init.pp as below helps to circumvent this issue,
command => "& "$file_path"",