saltstack-formulas / mysql-formula

Install the MySQL client and/or server

mysql-formula

Install the MySQL client and/or server on Linux and MacOS.

Table of Contents

• General notes
• Contributing to this repo
• Available states
  • mysql
  • mysql.macos
  • mysql.macos.remove
  • mysql.client
  • mysql.server
  • mysql.server_checks
  • mysql.disabled
  • mysql.database
  • mysql.python
  • mysql.user
  • mysql.remove_test_database
  • mysql.dev
  • mysql.repo
  • mysql.config
• Testing
  • Requirements
  • bin/kitchen converge
  • bin/kitchen verify
  • bin/kitchen destroy
  • bin/kitchen test
  • bin/kitchen login

General notes

See the full SaltStack Formulas installation and usage instructions.

If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section.

If you want to use this formula, please pay attention to the FORMULA file and/or git tag, which contains the currently released version. This formula is versioned according to Semantic Versioning.

See Formula Versioning Section for more details.

Contributing to this repo

Commit message formatting is significant!!

Please see How to contribute for more details.

Available states

• mysql
• mysql.macos
• mysql.macos.remove
• mysql.client
• mysql.server
• mysql.server_checks
• mysql.disabled
• mysql.database
• mysql.python
• mysql.user
• mysql.remove_test_database
• mysql.dev
• mysql.repo
• mysql.config

mysql

Meta-state including all server packages in correct order. This meta-state does not include mysql.remove_test_database.

mysql.macos

Install "MySQL Community Server", "MySQL Workbench", and other related mysql products on MacOS (and create Desktop shortcuts).

mysql.macos.remove

Remove "MySQL Community Server", "MySQL Workbench", and any other enabled products from MacOS.

mysql.client

Install the MySQL client package on Linux.

mysql.server

Install the MySQL server package and start the service.

Debian OS family supports setting MySQL root password during install via debconf.

Note

If no root password is provided in the pillar, a random one will be created. Because Hydrogen doesn't have easy access to a random function (test.rand_str isn't introduced until Helium), instead, we use the not-at-all random grains.server_id. As this is cryptographically insecure, future formula versions should use the newly available random.get_str method.

mysql.server_checks

Enforces a root password to be set.

mysql.disabled

Ensure that the MySQL service is not running.

mysql.database

Create and manage MySQL databases.

mysql.python

Install mysql python bindings.

mysql.user

Create and manage MySQL database users with definable GRANT privileges.

The state accepts MySQL hashed passwords or clear text. Hashed password have priority.

Note

See the salt.states.mysql_user docs for additional information on configuring hashed passwords.

Make sure to quote the passwords in the pillar so YAML doesn't throw an exception.

mysql.remove_test_database

Warning

Do not use this state if your MySQL instance has a database in use called test. If you do, it will be irrevocably removed!

Remove the database called test, normally created as part of a default MySQL installation. This state is not included as part of the meta-state above as this name may conflict with a real database.

mysql.dev

Install the MySQL development libraries and header files.

Note

Note that this state is not installed by the mysql meta-state unless you set your pillar data accordingly.

mysql.repo

Add the official MySQL 5.7 repository.

Note

Note that this state currently only supports MySQL 5.7 for RHEL systems. Debian and Suse support to be added. Also need to add the option to allow selection of MySQL version (5.6 and 5.5 repos are added but disabled) and changed enabled repository accordingly.

mysql.config

Manage the MySQL configuration.

Note

There are currently two common ways to configure MySQL, a monolithic configuration file or a configuration directory with configuration files per component. By default this state will use a configuration directory for CentOS and Fedora, and a monolithic configuration file for all other supported OSes.

Whether the configuration directory is used or not depends on whether mysql.config_directory is defined in the pillar. If it is present it will pick the configuration from individual component keys (mysql.server, mysql.galera, mysql.libraries, etc) with optional global configuration from mysql.global. The monolithic configuration, however, is defined separately in mysql.config.

Testing

Linux testing is done with kitchen-salt.

Requirements

• Ruby
• Docker

$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]

Where [platform] is the platform name defined in kitchen.yml, e.g. debian-9-2019-2-py3.

bin/kitchen converge

Creates the docker instance and runs the mysql main state, ready for testing.

bin/kitchen verify

Runs the inspec tests on the actual instance.

bin/kitchen destroy

Removes the docker instance.

bin/kitchen test

Runs all of the stages above in one go: i.e. destroy + converge + verify + destroy.

bin/kitchen login

Gives you SSH access to the instance for manual testing.

mysql-formula's Issues

Can not login using mysql after run

I can not log in to mysql after I used this formula and set the following in pillar:

I have put in fake pass words and logins in this file:

init_sls.txt

Otherwise Ive made no changes to the salt state files.

then ran it like so:
sudo salt [Server_IP_Address] state.apply mysql

Then I try to login from either the jump server or the server where mysql was installed it fails:
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on '[Server_IP_Address]' (111)

I'm assuming I have failed to set some needed value in the pillar.

Any help would be appreciated.
Jeff Wilson

in pillar.example lookup should be a child of server (and Amazon python package name is wrong/changed)

I ran into two related issue when trying to install mysql on an AWS instance.

1. The python package name needs to be MySQL-python26. That might be worth updating in the defaults for the formula.
2. When trying to override the python package via pillar us pillar.example for reference it was getting ignored. The pillar.example shows a wrong level of indentation for lookup, see below:

Current pillar.example:
mysql:
__server:
__lookup:
____python: python-mysqldb

Should be in pillar.example:
mysql:
__server:
____lookup:
______python: python-mysqldb

you can see from python.sls lookup is being referenced as a child of server:
{%- set mysql = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']%28'mysql:server:lookup'%29) %}

Edit: put underscores for spaces to show indentation in pillar.example snippets

Setting a "bind-address" pillar value doesn't restart MySQL

If I set in my pillar something like this:

mysql:
  server:
    mysqld:
      bind-address: 0.0.0.0

... then the configuration file gets properly updated and the server gets reloaded but it's actually not sufficient and MySQL continues to listen on its default localhost address.
It has to be manually restarted for the change to take effect.

That would be cool if the formula takes care of this by itself.

Add the skip-name-resolve option

skip-name-resolve is an important option but I am not sure how this one should configured from pillar.
What is the proper way to add this in pillar?

Issue with db schema load

Premise: Our saltmaster is stateless, so we can just delete the machine our autoscaling group will recreate it and all minions going to connect.

Our saltmaster will create the database, setup users and fetches the mysql dump from s3 and load it.

If i understand the SLS correctly schema load ({{ state_id }}_load) will always be triggered when the schema file is being handled ({{ state_id }}_schema). This is because of - watch: - file: {{ state_id }}_schema.

Is this expected behaviour? Once the database is created it should load the schema. It shouldn't load the schema file or execute another schema import.

Thoughts?

Missing debconf-utils package

After I implemented the mysql-formula, my vagrant provision was failing. giving me the following error:

----------
          ID: mysql_debconf
    Function: debconf.set
        Name: mysql-server
      Result: False
     Comment: State debconf.set found in sls mysql.server is unavailable
     Changes:  
----------

After some research I found that the debconf-utils package was missing on the vagrant box. After doing a sudo apt-get install debconf-utils the provision was working fine again. I think this formula is missing a dependency to this package.

one user, two hosts grants

Hi!
My mysql user "foo" can connect from localhost and from another ip. Same username, different ip.

At the moment I can't produce two entries in mysql.user table with the same "User" field and different "Host" field

Use map.jinja as defaults store

This issue is a "duplicate" of #43 as I have also the problem to install MariaDB on CentOS 7.

The mysql formula should use map.as defaults store with grains.filter_by mechanism to provide a layered defaults store.

I'm working on a patch on my repo https://github.com/refnode/mysql-formula.git and send a pull request soon.

How do you create a create a db, user and assign privileges?

I'm new to Salt and I'm having trouble using salt to create a db, user and assign privileges to my user. I've added the mysql directory from the mysql-formula to salt/roots/salt.

In the bottom of minion.conf I've added:

mysql.host: 'localhost'
mysql.port: 3306
mysql.user: 'root'
mysql.pass: 'somepass'
mysql.db: 'mysql'
mysql.unix_socket: '/tmp/mysql.sock'
mysql.charset: 'utf8'

In my mysql.sls for the host I'm provisioning using vagrant I have:

include:
  - mysql.server
  - mysql.client
  mysql_database:
    - present
    - name: intranet
  mysql_user:
    - present
    - name: intranet
    - password_hash: '*65F24BF0453608D25F9E6942F73F8182B90D3780'
    - require:
      - service: mysqld
  mysql_grants:
    - present
    - grant: all privileges
    - database: {{ pillar['mysql']['db'] }}.*
    - user: {{ pillar['mysql']['user'] }}
    - host: localhost

When I run vagrant provision I get:

[DEBUG   ] Rendered data from file: /srv/salt/intranet/mysql.sls:
include:
  - mysql.server
  - mysql.client
  mysql_database:
    - present
    - name: intranet
  mysql_user:
    - present
    - name: intranet
    - password_hash: '*65F24BF0453608D25F9E6942F73F8182B90D3780'
    - require:
      - service: mysqld
  mysql_grants:
    - present
    - grant: all privileges
    - database: intranet.*
    - user: intranet
    - host: localhost
[CRITICAL] Rendering SLS intranet.mysql failed, render error: while parsing a block collection
  in "<unicode string>", line 2, column 3:
      - mysql.server
      ^
expected <block end>, but found '?'
  in "<unicode string>", line 4, column 3:
      mysql_database:
      ^
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/salt/state.py", line 2095, in render_state
    sls, rendered_sls=mods
  File "/usr/lib/pymodules/python2.7/salt/template.py", line 74, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/pymodules/python2.7/salt/renderers/yaml.py", line 43, in render
    data = load(yaml_data, Loader=get_yaml_loader(argline))
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 393, in parse_block_sequence_entry
    "expected <block end>, but found %r" % token.id, token.start_mark)
ParserError: while parsing a block collection
  in "<unicode string>", line 2, column 3:
      - mysql.server
      ^
expected <block end>, but found '?'
  in "<unicode string>", line 4, column 3:
      mysql_database:
      ^
[DEBUG   ] Loaded no_out as virtual quiet
[DEBUG   ] Loaded json_out as virtual json
[DEBUG   ] Loaded yaml_out as virtual yaml
[DEBUG   ] Loaded pprint_out as virtual pprint
local:
    Data failed to compile:
----------
    Rendering SLS intranet.mysql failed, render error: while parsing a block collection
  in "<unicode string>", line 2, column 3:
      - mysql.server
      ^
expected <block end>, but found '?'
  in "<unicode string>", line 4, column 3:
      mysql_database:
      ^
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/salt/state.py", line 2095, in render_state
    sls, rendered_sls=mods
  File "/usr/lib/pymodules/python2.7/salt/template.py", line 74, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/pymodules/python2.7/salt/renderers/yaml.py", line 43, in render
    data = load(yaml_data, Loader=get_yaml_loader(argline))
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 393, in parse_block_sequence_entry
    "expected <block end>, but found %r" % token.id, token.start_mark)
ParserError: while parsing a block collection
  in "<unicode string>", line 2, column 3:
      - mysql.server
      ^
expected <block end>, but found '?'
  in "<unicode string>", line 4, column 3:
      mysql_database:
      ^

Obviously my mysql.sls isn't correct, any suggestions?

defaults.yaml OS support (OEL vs. RedHat)

We are using Oracle Enterprise Linux, which is basically the same as RedHat Enterprise Linux - the problem is that the "os" grain value is "OEL" for Oracle Linux, but the defaults.yaml is only prepared for RedHat (and CentOS and Fedora).

Duplicating the RedHat part in defaults.yaml and changing the key to "OEL" works fine.

My question is, could you include this in the formula?

Thanks in advance.

Add escape attribute

mysql_grants.present supports a boolean attribute to override the default escaping of string values. The formula should support this as well.

See luciddg#1

Installation of 'mysql-server' fails on systemd-based Ubuntu

Using the mysql-formula on Ubuntu >=15.04, using systemd, the installation of mysql-server will fail as described in Launchpad#1478849.

Still investigating workarounds for this issue. Will report here in case I find a way to get it working in a more-or-less hackish way.

Should be able to specify default collation for a new database

salt.modules.mysql.db_create has a collate attribute which allows to specify a default collation to be used when creating a new database.
It would be nice if this formula allows to specify it.

Problem: the current Pillar configuration doesn't allow to specify such kind of settings:

mysql:
  database:
    - foo
    - bar

as this is just a list of database names.

If you have a suggestion on how to configure this, I'm willing to submit a pull request for this feature.

If root_password is set to false under ubuntu, required debconf:mysql_debconf is not defined

To reproduce, set mysql:server:root_password to False in pillar and assert any state, for example mysql. salt reports The following requisites were not found: require: debconf: mysql_debconf

Cause:
server.sls def's mysql_debconf inside of {% if mysql_root_password %} so if root password is False, mysql_debconf is not defined. However, it is required by most states.

Can't use % in user:host

Using latest develop branch. Tried %, '%', "%" and same variants with *

mysql_user_0:
  mysql_user.present:
    - name: redacted
    - host: %    <======================
    - password: 'redacted'

    - connection_host: localhost
    - connection_user: root
    - connection_pass: 'redacted'

Database creation using incorrect name

The database.sls file is pre-pending "mysql_db" to the database name.

Just needs a name declaration added..
e.g.

• name: {{ database }}

mysql.client require {{ mysql.server }} package which is not needed on debian

part of mysql/config.sls included by mysql/client.sls, from here

mysql_config:
  file.managed:
    - name: {{ mysql.config.file }}
    - require:
      - pkg: {{ mysql.server }}
    - template: jinja
{% if "config_directory" in mysql %}
    - source: salt://mysql/files/my-include.cnf
{% else %}
    - source: salt://mysql/files/my.cnf
{% endif %}
    {% if os_family in ['Debian', 'Gentoo', 'RedHat'] %}
    - user: root
    - group: root
    - mode: 644
    {% endif %}

If I just want to install the client on a webserver, to probe mysql connection from the webserver to the dbserver, I don't need to have the server package to be installed.

On Debian jessie there'are, 2 packages available:

mariadb-common

dpkg -L mariadb-common
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/mariadb-common
/usr/share/doc/mariadb-common/copyright
/usr/share/doc/mariadb-common/changelog.Debian.gz
/etc
/etc/mysql
/etc/mysql/conf.d
/etc/mysql/conf.d/mariadb.cnf

mysql-common

root@web0:~# dpkg -L mysql-common
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/mysql-common
/usr/share/doc/mysql-common/NEWS.Debian.gz
/usr/share/doc/mysql-common/changelog.gz
/usr/share/doc/mysql-common/copyright
/usr/share/doc/mysql-common/changelog.Debian.gz
/etc
/etc/mysql
/etc/mysql/my.cnf
/etc/mysql/conf.d
/etc/mysql/conf.d/.keepme

my.cnf sections do not permit multiple entries of same type

my.cnf requires some parameters to be specified multiple times. An example would be binlog-ignore-db. This formula cannot support that due to the my.cnf sections in the pillar being a dict.

Using a list of dicts would solve this, but would break the current interface of the formula. Any guidance on how to resolve this?

mysql-formula fails on openSUSE (wrong key in defaults.yaml)

The mysql formula fails on openSUSE with

Rendering SLS 'production:mysql.server' failed: Jinja variable 'None' has no attribute 'server'
Rendering SLS 'production:mysql.server' failed: Jinja variable 'None' has no attribute 'server'

The reason is that default.yaml gets filtered by the 'os' grain - but doesn't include a matching key.

# salt '*' grains.get os
tux:   # openSUSE Tumbleweed
    openSUSE Tumbleweed
salt-test-1:   # openSUSE Leap 42.2
    SUSE

However default.yaml has "openSUSE" as a key, which doesn't match any of the above.

I can't speak for other distributions, but at least for openSUSE it would be a good idea to use the 'os_family' grain:

# salt '*' grains.get os_family
tux:  # openSUSE Tumbleweed
    Suse
salt-test-1:  # openSUSE Leap 42.2
    Suse

(CC @ccboltz)

Python package incorrect for CentOS 5

The python package specified in the map.jinja file is incorrect for CentOS 5.
I'm not sure if this also affects Redhat

The correct package name is 'python26-mysqldb'
Not sure of the best way to fix this.. Obviously the grain "osmajorrelease" will allow determination of the os version..
Is there a way to add a further condition to the grains.filter_by in the map?

Can't find "mysql/defaults.yaml" during mysql.server state

I can't install MySQL server because Jinja can't find mysql/defaults.yaml (related with this not solved but closed issue)

Currently I've tested this with a clean new VM deploying from my notebook (thanks to salt-ssh) without success.

I hope that this can be solved.

etc/salt/master

root_dir: .
file_roots:
  base:
    - srv/salt/states
    - srv/salt/formulas/mysql-formula
pillar_roots:
  base:
    - srv/salt/pillars

srv/salt/states/top.sls

base:
  '*':
    - mysql.server

srv/salt/pillars/top.sls

base:
  '*':
    - database

srv/salt/pillars/database.sls

mysql:
  server:
    root_user: 'root'
    root_password: 'pass'
    user: mysql
{% if grains['os_family'] == 'RedHat' %}
  lookup:
    server: mariadb-server
    client: mariadb
{% endif %}

Error:

          ID: mysql_config
    Function: file.managed
        Name: /etc/my.cnf
      Result: False
     Comment: Unable to manage file: Jinja error: mysql/defaults.yaml
              Traceback (most recent call last):
                File "/tmp/.root_b91e95_salt/salt/utils/templates.py", line 368, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/tmp/.root_b91e95_salt/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/tmp/.root_b91e95_salt/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 7, in top-level template code
                File "/tmp/.root_b91e95_salt/salt/utils/jinja.py", line 144, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: mysql/defaults.yaml

              ; line 7

              ---
              [...]
              # This config is generated by SALTSTACK
              # and all change will be overrided on next salt call
              {#-
              ===== FETCH DATA =====
              -#}
              {%- from "mysql/defaults.yaml" import rawmap with context -%}    <======================
              {%- from "mysql/supported_sections.yaml" import supported_sections with context -%}
              {%- set datamap = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) -%}
              {#-
              ===== COMBINE DATA =====
              -#}
              [...]
              ---
     Started: 05:25:44.050898
    Duration: 30.905 ms
     Changes:   

salt-ssh --versions-report

Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: Not Installed
           Mako: 1.0.0
         PyYAML: 3.11
          PyZMQ: 14.4.1
         Python: 2.7.9 (default, Apr  2 2015, 15:33:21)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 15.04 vivid
        machine: x86_64
        release: 3.19.0-30-generic
         system: Ubuntu 15.04 vivid

Multiple users with same name (different hosts)

Or be able to give 'hosts' as a list

mysql password leak in minion's log

when provisioning a mysql server, the root_password is written in clear in the minion's log, which is not so good...

 2015-06-08 18:27:33,049 [salt.state       ][INFO    ][22761] Running state [mysqladmin --user root password 'XXX'] at time 18:27:33.049208
 2015-06-08 18:27:33,050 [salt.state       ][INFO    ][22761] Executing state cmd.run for mysqladmin --user root password 'XXX'
 2015-06-08 18:27:33,051 [salt.loaded.int.module.cmdmod][INFO    ][22761] Executing command 'mysql --user root --password=\'XXX\' --execute="SELECT 1;"' in directory '/root'
 2015-06-08 18:27:33,129 [salt.loaded.int.module.cmdmod][INFO    ][22761] Executing command "mysqladmin --user root password XXX'" in directory '/root'
 2015-06-08 18:27:33,227 [salt.state       ][INFO    ][22761] {'pid': 23051, 'retcode': 0, 'stderr': '', 'stdout': ''}
 2015-06-08 18:27:33,227 [salt.state       ][INFO    ][22761] Completed state [mysqladmin --user root password 'XXX'] at time 18:27:33.227229

I'm not sure how to avoid that beside creating a custom module...

Adapt for MariaDB

It would be nice if you could adapt this to work with MariaDB - especially since CentOS/RedHat has dropped MySQL in favor of MariaDB.

debconf-utils should be checked if trying to set root passwd

Install fails on ubuntu (14.04) due to debconf-utils not being installed by default. It should install it (or at least have a toggle on/off)

mysql:server:root_user inconsistently used on Debian/Ubuntu

For Debian OS family, debconf is used to set root user password, which is fine, but root user cannot be specified this way thus it is set to root for all Debian systems.

When the formula attemps to do further changes to mysql like creating a database, etc, it fails if root_user is set to anything else than root.

Wrong CentOS/RHEL 7 defaults and state failure when mysql.version is not defined

Hey guys,

I'm opening this ticket to fix a couple of things I discovered when testing this formula on CentOS/RHEL 7.

I will be pushing a PR in a bit.

Thanks!

remove a mysql user functionality

Add a functionality to remove a database user:

Example pillar:

mysql:
  # Manage users
  # you can get pillar for existing server using scripts/import_users.py script
  user:
    frank:
      absent: true   ## <===== Add this, the user will be removed.
      password: 'somepass'
      host: localhost
      databases:
        - database: foo
          grants: ['select', 'insert', 'update']
        - database: bar
          grants: ['all privileges']

Run:

salt 'db*' state.apply mysql.user

Jinja variable 'dict object' has no attribute 'test.rand_str'

[CRITICAL] Rendering SLS "base:mysql" failed: Jinja variable 'dict object' has no attribute 'test.rand_str'
/srv/formulas/mysql-formula/mysql/database.sls(3):

---
{% from "mysql/map.jinja" import mysql with context %}

{% set mysql_root_pass = salt['pillar.get']('mysql:server:root_password', salt['test.rand_str'](64)) %}    <======================

The formula is the original one and it has no custom pillar data. It is a masterless setup with version 2014.1.6 salt-call.

Changing mysql root password on Ubuntu

After installing mysql-server with this formula, with a root_password set in pillar, I change the password in pillar and run state.highstate, but the root password in mysql stays the same. I see salt changing the mysql root password in debconf, but that is only used during install of the package.

Various configuration options for mysqld are missing

Unsure if this is intentional, but a variety of mysqld options are missing from the supported_params.yaml file, such as skip-networking and a whole bunch more on https://dev.mysql.com/doc/refman/5.7/en/security-options.html and https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html.

Is this by design? If it's just that options are missing and haven't been added, perhaps that second link's table could be trimmed down into YAML and included in the given file -- that would ensure all options are covered in one fell swoop.

formula does not set root password

When using the root_password option in the pillar.example to set the root password, I am unable to login as root using the root password:

sudo salt-call pillar.get mysql
local:
    ----------
    server:
        ----------
        root_password:
            xxx
        root_user:
            root

The login attempt:

mysql -u root -p
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

The output from debconf:

----------
          ID: mysql_debconf
    Function: debconf.set
        Name: mysql-server
      Result: True
     Comment:
     Started: 14:59:59.839264
    Duration: 1488.64 ms
     Changes:
              ----------
              mysql-server/root_password:
                  (password hidden)
              mysql-server/root_password_again:
                  (password hidden)
              mysql-server/start_on_boot:
                  true

mysql_config require package mysql-server: problem when specifying a log file size

Hello,

I'm using this formula with an option to change the InnoDB log file size.

innodb_log_file_size: 256M

It causes problem on my Debian VM: mysql is installed, configured but can't (re)start. The error tells about the current log file size which is different from the one specified in my.cnf.

I think what happens is the following:

• packages installation: Debian creates a default my.cnf, and then starts mysql server
• creation of the /etc/mysql/my.cnf
• restart of the mysql service: this is where it fails

I modified the formula to make mysql_config state run before the mysql.server package installation, and it solved my problem.

What do you think about this? Should it be possible, and safe, to change the states order?

Thanks.

mysql/defaults.yaml not found

I'm getting the following error when I run the mysql state:

----------
          ID: mysql_config
    Function: file.managed
        Name: /etc/mysql/my.cnf
      Result: False
     Comment: Unable to manage file: Jinja error: mysql/defaults.yaml
              Traceback (most recent call last):
                File "/tmp/.root_4e6e95__salt/salt/utils/templates.py", line 306, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/tmp/.root_4e6e95__salt/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/tmp/.root_4e6e95__salt/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 7, in top-level template code
                File "/tmp/.root_4e6e95__salt/salt/utils/jinja.py", line 143, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: mysql/defaults.yaml

              ; line 7

              ---
              [...]
              # This config is generated by SALTSTACK
              # and all change will be overrided on next salt call
              {#-
              ===== FETCH DATA =====
              -#}
              {%- from "mysql/defaults.yaml" import rawmap with context -%}    <======================
              {%- from "mysql/supported_sections.yaml" import supported_sections with context -%}
              {%- set datamap = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) -%}
              {#-
              ===== COMBINE DATA =====
              -#}
              [...]
              ---
     Started: 16:42:01.829849
    Duration: 40.14 ms
     Changes:

Fails if FQDN is localhost or localhost.localdomain

> salt-call state.apply mysql.server  
[CRITICAL] Rendering SLS 'base:mysql.server' failed: Conflicting ID 'mysql_delete_anonymous_user_localhost.localdomain'
local:
    Data failed to compile:
----------
    Rendering SLS 'base:mysql.server' failed: Conflicting ID 'mysql_delete_anonymous_user_localhost.localdomain'

See mysql/server.sls:

{% for host in ['localhost', 'localhost.localdomain', salt['grains.get']('fqdn')] %}
mysql_delete_anonymous_user_{{ host }}:
  mysql_user:
    - absent
    - host: {{ host or "''" }}
    - name: ''
    - connection_host: '{{ mysql_host }}'
    - connection_user: '{{ mysql_salt_user }}'
    {% if mysql_salt_password %}
    - connection_pass: '{{ mysql_salt_password }}'
    {% endif %}
    - connection_charset: utf8
    - require:
      - service: mysqld
      - pkg: mysql_python
      {%- if (mysql_salt_user == mysql_root_user) and mysql_root_password %}
      - cmd: mysql_root_password
      {%- endif %}
{% endfor %}

Non-standard 'datadir' needs AppArmor changes on Ubuntu

When using a non-standard datadir on Ubuntu, changes to AppArmor are required, otherwise MySQL won't start anymore.
See also: https://blogs.oracle.com/jsmyth/entry/apparmor_and_mysql

Similar changes are most likely required for SELinux enabled systems.

Automated testing feature

Hey guys,

I was working on this formula and wanted to have a way to test in multiple OS at once so I come up with a solution that involves TravisCI and KitchenCI.

I tried to draft a global solution that can be easily ported to other formulas because it reads the FORMULA file to get the formula name and based on that knows the name of the state to execute.

At this time for this particular formula it tests on the following OS:

• Ubuntu 12.04 (Precise)
• Ubuntu 14.04 (Trusty)
• Ubuntu 16.04 (Xenial)
• Debian 7 (Wheezy)
• Debian 8 (Jessie)

Once #147 is fixed we can add support for CentOS and Fedora, and once test-kitchen/kitchen-docker#156 is merged we will be able to support openSuSE too.

I think this feature will allow contributors to test their commits on multiple scenarios to ensure formula quality.

Let me know if you have any question/improvement!

Thanks,
Andres

Setup from scratch needs two runs

When setting up MySQL from scratch (without a pre-existing /var/lib/mysql/), I get errors like

          ID: mysql_delete_anonymous_user_localhost.localdomain
    Function: mysql_user.absent
        Name:
      Result: False
     Comment: MySQL Error 1045: Access denied for user 'salt'@'localhost' (using password: YES)

This happens because mysql_delete_anonymous_user_* runs before the salt_user gets created.

Additionally, mysql.salt-user isn't included in init.sls, which means by default salt_user doesn't get created at all.

Proof of concept-patch:

--- a/mysql/init.sls
+++ b/mysql/init.sls
@@ -11,6 +11,7 @@
 
 include:
   - mysql.server
+  - mysql.salt-user
   - mysql.database
   - mysql.user
 {% if mysql_dev %}

--- a/mysql/server.sls
+++ b/mysql/server.sls
@@ -52,6 +52,7 @@ mysql_delete_anonymous_user_{{ host }}:
     {% endif %}
     - connection_charset: utf8
     - require:
+      - mysql_root_password
       - service: mysqld
       - pkg: mysql_python
       {%- if (mysql_salt_user == mysql_root_user) and mysql_root_password %}

Note that this patch is not perfect - requiring mysql_root_password delays mysql_delete_anonymous_user_* enough to get it working. However it uses the mysql_salt_user, not root - so the dependency on mysql_root_password is technically wrong ;-)

(CC @ccboltz)

Deprecated iteritems() method not forward-compatible with Python 3

Formulas iterating over dictionaries should use the items() method instead of iteritems() because the latter method has been removed from Python 3. While the items() method in Python 2 uses more memory than iteritems(), these dictionaries are typically small (e.g., sets of settings from Pillar).

mysql.server failing to render

On a Debian VM, Salt is printing the below messages:
[ERROR ] Rendering exception occurred: Jinja variable 'dict object' has no attribute 'version' [CRITICAL] Rendering SLS 'base:mysql.server' failed: Jinja variable 'dict object' has no attribute 'version' [ERROR ] Data passed to highstate outputter is not a valid highstate return: {'local': ["Rendering SLS 'base:mysql.server' failed: Jinja variable 'dict object' has no attribute 'version'"]}

The pillar.example doesn't include a 'version' entry, nor does the defaults.yaml . The 'version' value is only noted on lines 67, 91, and 108 of server.sls . Further, should lines 91 and 108 be
{% if ( os_family == 'RedHat' or os_family == 'Suse' ) and mysql.version >= 5.7 %}
or
{% if os_family == 'RedHat' or ( os_family == 'Suse' and mysql.version >= 5.7 ) %}
instead? As currently written, it appears to cause a check of mysql.version regardless of the OS family.

pillar lookup.server value is ignored

In Ubuntu 14.04 i'm setting lookup.server value in my pillar file as: mysql-server-5.6. This value is shown with the pillar.tems. However running salt-call --local state.highstate, installs the value from the defaults.yml, which is 'mysql-server'.
Changing the value in the defaults.yml, to mysql-server-5.6 works and the correct version gets installed.

Add option to specify bind interface instead of bind ip

With multiple mysql servers to setup it would be more useful to be able to specify interface service is listening on then ip address, as it can be easily obtained in runtime from specified interface.

Database names are prefixed with mysql_db_

I am not sure if this is intended but it can be fixed by adding name to the database state

Changing 'innodb_log_file_size' needs manual intervention

When changing innodb_log_file_size, mysqld won't start anymore without moving /var/lib/mysql/ib_logfile* out of the way.

See also this article for details.

I think it could be basically implemented like this:

{% for section, data in mysql.config.sections.iteritems() %}
  {% if 'innodb_log_file_size' in data %}
    {% set innodb_log_file_workaround = true %}
  {% endif %}
{% endfor %}
{% if innodb_log_file_workaround is defined and innodb_log_file_workaround is sameas true %}
mysqld-stopped-for-logfile-workaround:
  service.dead:
    - name: {{ mysql.service }}
    - prereq:
      - module: innodb-logfile-workaround
    - require:
      - pkg: mysqld

  {% set curtime = None | strftime("%Y-%m-%d-%H:%M:%S") %}
  {% set backup_dir = mysql.config.sections.mysqld.datadir ~ '/../ib_logfile-backup-' ~ curtime ~ '/' %}
innodb-logfile-backupdir:
  file.directory:
    - name: {{ backup_dir }}
    - makedirs: true

innodb-logfile-workaround:
  module.run:
    - name:   cmd.run
    - cmd:    "mv {{ mysql.config.sections.mysqld.datadir }}/ib_logfile* {{ backup_dir }}"
    - onlyif: "ls {{ mysql.config.sections.mysqld.datadir }}/ib_logfile*"
    - prereq:
      - file: mysql_config
    - require:
      - file: innodb-logfile-backupdir
{% endif %}

This is not fully working yet and a bit hackish - other ideas/sugestions/PRs welcome!

Adapt for number of mysql server instances on one minion

Using this formula it's only possible to configure one mysql server per minion.
Does anyone have ideas of how to simply adapt it for several ones per minion?

unable to start mysql 5.7

i noticed something interesting with mysql 5.7:
in the init script, it looks for the pid file by searching for "pid-file" in the output of my_print_defaults.
but the mysql-formula doesn't set that option. it sets "pid_file" and by default it only sets that option for the [mysqld] section and not for [mysqld_safe]. mysqld doesn't care but the init script waits forever. the fallback to $MYSQLDATA/$(hostname).pid doesn't work as well because the pid-file is located at /var/run/mysqld/mysqld.pid

/etc/init.d/mysql:

...
get_mysql_option() {
    RESULT=$(my_print_defaults "$1" | sed -n "s/^--$2=//p" | tail -n 1)
    if [ -z "$RESULT" ];
    then
        RESULT="$3"
    fi
    echo $RESULT
}

get_running () {
    PIDFILE=$(get_mysql_option mysqld_safe pid-file "")
    if [ -z "$PIDFILE" ];
    then
        PIDFILE=$(get_mysql_option mysqld pid-file "$MYSQLDATA/$(hostname).pid")
    fi
    if [ -e "$PIDFILE" ] && [ -d "/proc/$(cat "$PIDFILE")" ];
    then
        echo 1
    else
        echo 0
    fi
}
...

Unable to configure the MySQL server through MySQL Installer

Hi There,
i am trying to configure MySQL Server through MySQL installer. After Clicking on re-configure option, the link leads to page as shown in attachment. But the page doesn't have any next/Ok option below. I am Stuck at this point.

Pls Share your insights!

produce managed file with mysql root acces from salt module related to pillar root_password

When you install this formula you may expect that the defined password for mysql's root on the managed server will be available on the salt master for mysql.module call:

salt 'db*' mysql.db_list

salt 'db*' mysql.user_create bob 10.0.0.3 pipo

# etc.

It requires to have setup in /etc/salt/minion as said in the doc above:

mysql.host: 'localhost'
mysql.port: 3306
mysql.user: 'root'
mysql.pass: 'S3retPass4root!'
mysql.db: 'mysql'
mysql.unix_socket: '/tmp/mysql.sock'
mysql.charset: 'utf8'

or

mysql.default_file: '/etc/mysql/debian.cnf'

I propose to use the second form in the file /root/.my.cnf

I accomplished a prototype here: server.sls

I will post more in this ticket about security. And sum up discussion in #salt IRC channel.

Handling of non-standard `datadir`

Right now, there's no handling at all for non-default locations of datadir.

• a fresh deployment doesn't create the datadir
• changing an existing deployment doesn't move the existing datadir to its new location (shutdown of mysql service, moving datadir, starting mysql service again)

There might be some difficulties involved in this, as the package management of some distributions adds mechanisms (postinst etc.) to the MySQL package which expects the default path (/var/lib/mysql) and otherwise might break the package installation/always re-create the datadir, etc.

User grant fails to execute

Since 04c8db5 I'm seeing the following error while using this formula to build MariaDB on CentOS 7. The user grant appears to fail while applying a minion highstate but running the command manually works fine.

----------
          ID: mysql_user_user_localhost_0
    Function: mysql_grants.present
        Name: admin_database_all
      Result: False
     Comment: Failed to execute: "GRANT all privileges ON database.* TO user@localhost" (MySQL Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1)
     Started: 23:11:53.928118
    Duration: 5.062 ms
     Changes:   

# mysql -psecret
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 127
Server version: 5.5.50-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use database
Database changed
MariaDB [database]> GRANT all privileges ON database.* TO user@localhost;
Query OK, 0 rows affected (0.03 sec)

MariaDB [database]>

Pillar data:

mysql:
  server:
    root_user: 'root' 
    root_password: 'secret'
    mysqld:
      bind-address: localhost
      log_error: /var/log/mariadb/mariadb.log
      pid-file: /var/run/mariadb/mariadb.pid
    mysqld_safe:
      bind-address: localhost
      log_error: /var/log/mariadb/mariadb.log
      pid-file: /var/run/mariadb/mariadb.pid
  database:
    - database
  schema:
    database:
      load: True
      source: salt://files/database.sql
  lookup:
    server: mariadb-server
    client: mariadb
    service: mariadb
  user:
    user:
      password: 'secret'
      host: localhost
      databases:
        - database: database
          grants: ['all privileges']

Master:

# salt --versions-report
Salt Version:
           Salt: 2016.3.2

Dependency Versions:
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.21.0
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
         Python: 2.7.5 (default, Aug 18 2016, 15:58:25)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.18.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

Minion:

$ salt-call --versions-report
Salt Version:
           Salt: 2016.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug 18 2016, 15:58:25)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.22.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core