GithubHelp home page GithubHelp logo

claranet / puppet-varnish Goto Github PK

View Code? Open in Web Editor NEW
7.0 14.0 62.0 284 KB

Puppet module to install and configure Varnish cache

Home Page: https://forge.puppet.com/claranet/varnish

License: Apache License 2.0

Ruby 43.92% Puppet 42.07% HTML 12.42% Dockerfile 1.58%

puppet-varnish's Introduction

puppet-varnish

Build Status Puppet Forge Forge Downloads

Table of Contents

  1. Overview - What is the puppet-varnish module?
  2. Module Description - What does the module do?
  3. Setup - The basics of getting started with puppet-varnish
  4. Usage - Configuration options and additional functionality
  5. Limitations - OS compatibility, etc.
  6. Development - Guide for contributing to the module

Overview

This module Installs and configures Varnish.

Puppet 3 Support

Please note that the master branch of this module does not support Puppet 3!

On 31st December 2016, support for Puppet 3.x was withdrawn. As such, this module no longer supports Puppet 3 - if you require Puppet 3 compatibility, please use the latest version 4.x version from the Puppet Forge, or the puppet3 branch in Git.

Module Description

This module Supports Varnish versions 3.0, 4.0, 4.1, 5.0, 5.1, 5.2, 6.0, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6 across Ubuntu 14.04/16.04/18.04, Debian 7/8/9/10 and RedHat derivates 6/7.

This module will install Varnish, by default version 4.1 from the official Packagecloud repositories, adding EPEL for RedHat-like systems and working around a SELinux policy bug in RHEL/CentOS 6 for Varnish 4.0 and above.

It will also install and configure a Systemd service for certain OS/Varnish combinations.

If necessary, you can specify any of the Varnish versions above, although there are imcompatibilities with some versions of Varnish and some OS versions, see Limitations.

Setup

To accept all default parameters - at minimum it is suggested you set a secret (if not explicitly set, one will be created via /proc/sys/kernel/random/uuid) and overwrite the packaged default.vcl.

  class { '::varnish':
    secret => '6565bd1c-b6d1-4ba3-99bc-3c7a41ffd94f',
  }

  ::varnish::vcl { '/etc/varnish/default.vcl':
    content => template('data/varnish/default.vcl.erb'),
  }

Multiple Listen Interfaces

Varnish supports listening on multiple interfaces. The module implements this by exposing a listen parameter, which can either be set to a String value for one interface (e.g. 127.0.0.1 or 0.0.0.0), or an array of values.

By default, the module will append listen_port to each element of the array - however to set a different port for each interface, just append it using standard notation, for example: 127.0.0.1:8080.

Usage

Examples

To use a static file with varnish::vcl rather than a template:

  ::varnish::vcl { '/etc/varnish/default.vcl':
   content => file('data/varnish/default.vcl'),
   # Equivalent to: source => 'puppet:///modules/data/varnish/default.vcl'
  }

To pin Varnish to a specific version - you may also provide varnish_version as long as it matches the major and minor version in package_ensure, however the module will automatically calculate varnish_version if not set:

  class { '::varnish':
    package_ensure => '4.0.5-1~xenial',
  }

To configure Varnish to listen on port 8080 on localhost and port 6081 on 172.16.100.10:

  class { '::varnish':
    listen => ['127.0.0.1:8080','172.16.100.10:6081'],
  }

To configure Varnish to listen on port 80, specifically on localhost and 192.168.1.195:

  class { '::varnish':
    listen      => ['127.0.0.1','192.168.1.195'],
    listen_port => '80',
  }

To use multiple storage backends in varnish for example a primary 4GB memory backend and a 50GB file backend:

  class { '::varnish':
    storage_type => 'malloc',
    storage_size => '4G',
    storage_additional => [
      'file,/var/lib/varnish/varnish_additional.bin,50G',

    ]
  }

Parameter Reference

Parameter Description
addrepo Whether to add the official Varnish repos
varnish_version Major Varnish version
secret Secret for admin access
secret_file File to store the secret
vcl_conf Varnish vcl config file path
listen IP to bind to
listen_port TCP port to listen on
admin_listen Admin IP to bind to
admin_port TCP port for admin to listen on
min_threads Minimum Varnish worker threads
max_threads Maximum Varnish worker threads
thread_timeout Terminate threads after this long idle
storage_type malloc or file
storage_file File to mmap on disk for cache storage
storage_size Size of storage file or RAM, eg 10G or 50%
storage_additional Hash of additional storage backends, passed plainly to varnishd -s after the normal configured storage backends
package_ensure Version of Varnish package to install, eg 3.0.5-1.el6
runtime_params hash of run-time parameters to be specified at startup

Limitations

There are several limitations with various Varnish and OS combinations. The module will attempt to flag known issues, however:

  • Varnish 3.0 is not supported on Ubuntu 16.04
  • Varnish 5.x, 6.x, supports only Debian 8 and Ubuntu 16.04

Development

  • Copyright (C) 2017 Claranet
  • Distributed under the terms of the Apache License v2.0 - see LICENSE file for details.

puppet-varnish's People

Contributors

craigwatson avatar dougneal avatar fe80 avatar joekohlsdorf avatar kidswiss avatar l-walton avatar lpmi-13 avatar mwilkinson avatar niteman avatar ralfbosz avatar rodricels avatar russwhelan avatar smerrill avatar steve-stonehouse avatar talisto avatar zicklam avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-varnish's Issues

Cannot provision puppet-varnish in vagrant

My Puppetfile initializes the puppet-varnish module:

...
## Install Module: varnish
mod 'varnish',
  :git => '[email protected]:BashtonLtd/puppet-varnish.git',
  :ref => '3.0.0'
...

Then, the install.pp definition installs the module:

### Note: the prefix 'cache::', corresponds to a puppet convention:
###
###       https://github.com/jeff1evesque/machine-learning/issues/2349
###
class cache::varnish::install {
    class { 'varnish': }
}

Finally, my configure.pp:

### Note: the prefix 'cache::', corresponds to a puppet convention:
###
###       https://github.com/jeff1evesque/machine-learning/issues/2349
###
class cache::varnish::configure {
    ## variables
    $config_file = 'cache/default.vcl'

    ## configure vcl
    varnish::vcl { '/etc/varnish/default.vcl':
        content => file(dos2unix($config_file)),
    }
}

implements a corresponding default.vcl.erb, which is required by varnish. However, when I run the above puppet definitions (via vagrant up), I get the following error traceback:

$ vagrant up
...
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment development...
==> default: Warning: ModuleLoader: module 'varnish' has unresolved dependencies
 - it will only see those that are resolved. Use 'puppet module list --tree' to
see information about modules
==> default: Warning: ModuleLoader: module 'varnish' has unresolved dependencies
 - it will only see those that are resolved. Use 'puppet module list --tree' to
see information about modules
==> default: Warning: ModuleLoader: module 'varnish' has unresolved dependencies
 - it will only see those that are resolved. Use 'puppet module list --tree' to
see information about modules
==> default: Warning: ModuleLoader: module 'varnish' has unresolved dependencies
 - it will only see those that are resolved. Use 'puppet module list --tree' to
see information about modules
==> default: Warning: ModuleLoader: module 'varnish' has unresolved dependencies
 - it will only see those that are resolved. Use 'puppet module list --tree' to
see information about modules
==> default: Notice: Compiled catalog for drupal-demonstration.com in environmen
t development in 2.52 seconds
==> default: Notice: /Stage[main]/Varnish::Repo::El7/Yumrepo[varnish-cache]/ensu
re: created
==> default: Notice: /Stage[main]/Varnish::Install/Package[varnish]/ensure: crea
ted
==> default: Notice: /Stage[main]/Varnish::Secret/File[/etc/varnish/secret]/selu
ser: seluser changed 'unconfined_u' to 'system_u'
==> default: Notice: /Stage[main]/Varnish::Config/File[/etc/varnish/varnish.para
ms]/content: content changed '{md5}73adaa45730eaad22aa78a77cd7c0554' to '{md5}bb
8b07352b460b9f8b17d117a192ad22'
==> default: Notice: /Stage[main]/Varnish::Service/Service[varnish]/ensure: ensu
re changed 'stopped' to 'running'
==> default: Notice: /Stage[main]/Cache::Varnish::Configure/Varnish::Vcl[/etc/va
rnish/default.vcl]/File[/etc/varnish/default.vcl]/content: content changed '{md5
}43f33efc867d0ecd6f2ab08878a936bb' to '{md5}f6a26098425c6ad4d2b34c8fb47d05ab'
==> default: Notice: /Stage[main]/Varnish::Service/Exec[vcl_reload]/returns: /et
c/varnish/varnish.params: line 9: $'\r': command not found
==> default: Notice: /Stage[main]/Varnish::Service/Exec[vcl_reload]/returns: /et
c/varnish/varnish.params: line 11: $'\r': command not found
==> default: Notice: /Stage[main]/Varnish::Service/Exec[vcl_reload]/returns: /et
c/varnish/varnish.params: line 15: $'\r': command not found
==> default: Notice: /Stage[main]/Varnish::Service/Exec[vcl_reload]/returns: Err
or: RELOAD_VCL is not set to 1
==> default: Error: /Stage[main]/Varnish::Service/Exec[vcl_reload]: Failed to ca
ll refresh: /usr/sbin/varnish_reload_vcl returned 2 instead of one of [0]
==> default: Error: /Stage[main]/Varnish::Service/Exec[vcl_reload]: /usr/sbin/va
rnish_reload_vcl returned 2 instead of one of [0]
==> default: Notice: Applied catalog in 11.88 seconds
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

puppetlabs/apt old

Hello,

It appears that while you define support for stdlib < 6.0 your current version of puppetlabs/apt is at < 5.0 which resolves in the latest of 4.5.1 https://forge.puppet.com/puppetlabs/apt/4.5.1/dependencies which has stdlib at < 5.0 meaning I currently can't use it with my other dependencies.

Is there any hope on getting this updated to the latest puppetlabs/apt?

Add install_options to package spec(?)

Hi, have been having major headaches with installing on a CentOS server where an older version of varnish is in one of the repositories. Finally got round it by patching your module to add install options on the varnish package spec, and then passing a parameter to disable the repo in question.

I don't know if that's the best approach: any other ideas would be welcome. I'm fairly new to Puppet. Puppet version is 3.8.

systemd file on Debian Buster not working

My Configuration:

varnish::addrepo: false
varnish::listen:

  • ':80
    else are defaults.

The Systemd-Service file is generated with an "" at the end of the Service section, this breaks the Startup of the varnish service with:
Oct 17 08:03:19 web01p varnishd[16430]: Error: Too many arguments ([Install]...)
Oct 17 08:03:19 web01p varnishd[16430]: (-? gives usage)

varnish.service.txt

Not updating systemd service file

In case systemd is used (debian 8 for ex), the /etc/default/varnish is not used anymore, and the start line is hardcoded in systemd :

grep Start= /etc/systemd/system/multi-user.target.wants/varnish.service
ExecStart=/usr/sbin/varnishd -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

So this need to be updated in place of /etc/default/varnish

Config files must be of the same version of installed Varnish package

EDIT: this seems to only apply when $varnish::addrepo = false. Otherwise, this depends on whether the installer picks the package from Varnish repo or the distro ones.

Variable $varnish::params::package_ensure is set to present, causing Varnish to be installed in the latest version available. Variable $varnish::params::varnish_version is set to '3.0' or '4.0', depending on distro and version, so determining the version of the config files. This causes Varnish package and config files to have different versions in some situations. One example I hit in my tests is RHEL 6, which had Varnish installed with version 4 and configured with version 3 files.

As a workaround, I set $varnish::varnish_version to '4.0' in Hiera.

Suggestions:

  • Class varnish::config could query the installed Varnish package version via a custom fact
  • Variable $varnish::varnish_version could be set by checking the available installable version (like via yum list)

Support for IPv6 or multiple listen addresses

Hi,

Currently the module has no support for ipv6 or multiple listen addresses. Ideal this would de implemented as a array in daemon options or as new config variable varnish.params

Debian 8 doesn't use systemd

With a Debian Jessie, the module use sysvinit instead of systemd.

Puppet says that $::lsbdistrelease (8) is lower than 8.0: versioncmp(8, 8.0) == -1

case $::operatingsystem {
'Ubuntu': {
$systemd_version = '16.04'
}
'Debian': {
$systemd_version = '8.0'
}
default: {
fail("Unsupported Debian OS: ${::operatingsystem}")
}
}
if versioncmp($::lsbdistrelease,$systemd_version) >= 0 {
$os_service_provider = 'systemd'
} else {
$os_service_provider = 'sysvinit'
}

Pull request #51

Fixing deprecation issues

Currently, Puppet is throwing deprecation errors when using the Varnish module on Puppet 5. The Varnish module is still using, for example, validate_string, which has been deprecated.

Would like to manage varnishncsa service and its logformat.

Affected Puppet, Ruby, OS and module versions/distributions

  • Puppet: 4.10
  • Ruby:
  • Distribution: Ubuntu 14.04 & 18.04
  • Module version: 5.0.1

What are you seeing

I'm in the process of replacing our Ubuntu 14.04 servers with 18.04 and want to use this module instead of a legacy one we built many years ago.

I can manage Varnish just fine, but am unable to manage the varnishncsa logging service.

What behaviour did you expect instead

I'd like to be able to manage and configure the varnishncsa service via this module. It is included with Varnish but disabled by default. So that means I'd like to define the logformat it uses via Hiera, as well as ensuring the service is started and running.

How do you currently run varnishncsa, or do you not log Varnish requests at all? Would it be possible to add this functionality to your otherwise excellent module?

I currently define the logformat on Ubuntu 14.04 in /etc/default/varnishncsa (yes, all the backslashes are necessary to get json output). On systemd systems such as 18.04 this should probably be done differently.

VARNISHNCSA_ENABLED=1
LOG_FORMAT="-F{\\\"timestamp\\\": \\\"%{%Y-%m-%dT%T%z}t\\\",\\\"client_ip\\\": \\\"%h\\\",\\\"varnish.bytes_write\\\": %b,\\\"varnish.hit_miss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnish.time_first_byte\\\": %{Varnish:time_firstbyte}x,\\\"varnish.handling\\\": \\\"%{Varnish:handling}x\\\",\\\"varnish.cacheable\\\": \\\"%{X-Cacheable}o\\\",\\\"varnish.response_time_usec\\\": %D,\\\"content_type\\\": \\\"%{Content-Type}o\\\",\\\"x_forwarded_for\\\": \\\"%{X-Forwarded-For}i\\\",\\\"auth\\\": \\\"%u\\\",\\\"status_code\\\": %s,\\\"http.request_full\\\": \\\"%r\\\",\\\"http.request\\\": \\\"%U\\\",\\\"http.querystring\\\": \\\"%q\\\",\\\"http.host\\\": \\\"%{host}i\\\",\\\"http.method\\\": \\\"%m\\\",\\\"http.referrer\\\": \\\"%{Referer}i\\\",\\\"useragent\\\": \\\"%{User-agent}i\\\"}"

This file is sourced by the init script /etc/init.d/varnishncsa and the LOG_FORMAT variable is passed to the daemon.

start_varnishncsa() {
<--snip-->
    if start-stop-daemon --start --pidfile ${PIDFILE} \
        --chuid $USER --exec ${DAEMON} -- ${DAEMON_OPTS} \
        "${LOG_FORMAT}" \
        > ${output} 2>&1; then
<--snip-->

Much appreciated!

Ubuntu 16.04 support?

1) copy /lib/systemd/system/varnish.service to /etc/systemd/system/

It will need configuration for port binding etc:

cat /lib/systemd/system/varnish.service

[Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/4.1/ man:varnishd

[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/reload-vcl
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

[Install]
WantedBy=multi-user.target

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.