Can't find "mysql/defaults.yaml" during mysql.server state about mysql-formula

[ogabrielsantos]

Editing /etc/salt/master solve this odd issue:

extra_filerefs:
  - salt://mysql/defaults.yaml
  - salt://mysql/supported_sections.yaml

Maybe documentation could be improved or templating file inclusion avoided in order to prevent future issues to be opened.

If there are no new considerations this issue can be closed. Thanks.

[ogabrielsantos]

Tried with different salt version without success:

                  Salt: 2015.5.0
                Python: 2.7.9 (default, Apr  2 2015, 15:33:21)
                Jinja2: 2.7.3
              M2Crypto: Not Installed
        msgpack-python: 0.4.2
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.11
                 ioflo: Not Installed
                 PyZMQ: Not Installed
                  RAET: Not Installed
                   ZMQ: Not Installed
                  Mako: 1.0.0
 Debian source package: 2015.5.0+ds-1utopic1

[gravyboat]

I'll try to take a look at this sometimes this week, @iggy this is similar to the other issue with this same problem, however @ogabrielsantos hasn't tried workarounds so we have a clean slate, any suggestions/insight?

[puneetk]

change the file root ?

root_dir: .
file_roots:
base:
- srv/salt/states
- srv/salt/formulas
pillar_roots:
base:
- srv/salt/pillars

[ogabrielsantos]

@puneetk srv/salt/formulas/mysql-formula refer to a git module clone inside formulas. Changing to srv/salt/formulas will invalidate the formula/state.

├── srv
│   └── salt
│       ├── formulas
│       │   └── mysql-formula
│       │       ├── FORMULA
│       │       ├── import_users.py
│       │       ├── LICENSE
│       │       ├── mysql
│       │       │   ├── client.sls
│       │       │   ├── database.sls
│       │       │   ├── defaults.yaml
│       │       │   ├── dev.sls
│       │       │   ├── disabled.sls
│       │       │   ├── files
│       │       │   │   ├── my.cnf
│       │       │   │   └── usr-my.cnf
│       │       │   ├── init.sls
│       │       │   ├── python.sls
│       │       │   ├── remove_test_database.sls
│       │       │   ├── server.sls
│       │       │   ├── supported_sections.yaml
│       │       │   └── user.sls
│       │       ├── pillar.example
│       │       ├── README.rst
│       │       └── scripts
│       │           └── import_users.py
│       ├── pillars
│       │   ├── database.sls
│       │   └── top.sls
│       └── states
│           └── top.sls

[gravyboat]

I find it interesting that this issue exists: saltstack/salt#21370. @ogabrielsantos can you spin a VM and install a minion then try from there? So via salt itself, and not salt-ssh?

[ogabrielsantos]

I'll try as soon as possible @gravyboat

I don't think that this is a issue related to mysql-formula. Tested a sample state with same error:

etc/salt/master

root_dir: .
file_roots:
  base:
    - srv/salt/states

srv/salt/states/top.sls

base:
  '*':
    - sample

srv/salt/states/sample/init.sls

{% from "sample/sample.jinja" import sampledata with context %}

{{ sampledata.file }}:
  file.managed:
    - source: salt://sample/sample.cnf
    - template: jinja

srv/salt/states/sample/sample.cnf

{% from "sample/sample.jinja" import sampledata with context %}

srv/salt/states/sample/sample.jinja

{% load_yaml as rawmap %}
RedHat:
  file: /etc/my.cnf
{% endload %}

{% set sampledata = rawmap.get(grains.os_family) %}

[gravyboat]

@ogabrielsantos Interesting, thanks for testing that. I'm headed out for the evening, let me know what it looks like when you're able to test with the minion.

[ogabrielsantos]

Currently confirmed that CentOS 6.7 (minion) works fine with a CentOS 7 (master) using both salt and salt-ssh. Maybe it is a Saltstack dependency related issue?

Minion

[root@li1289-145 ~]# salt-minion --versions-report
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: unknown
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 4.1.5-x86_64-linode61
         system: CentOS 6.7 Final

Master

[root@li1278-54 sample]# salt-ssh --versions-report
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.1.1503 Core
        machine: x86_64
        release: 4.1.5-x86_64-linode61
         system: CentOS Linux 7.1.1503 Core

[ogabrielsantos]

I've patched this workaround which remove extra_filerefs and jinja templating file inclusion requirements as per @basepi comment

From f42f48666503a970636fd9c06be5dd350be889c5 Mon Sep 17 00:00:00 2001
From: Gabriel Santos <[email protected]>
Date: Mon, 28 Sep 2015 18:24:42 -0300
Subject: [PATCH] Workaround Jinja extra_filerefs requirement

---
 mysql/files/my.cnf | 6 ------
 mysql/server.sls   | 4 ++++
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/mysql/files/my.cnf b/mysql/files/my.cnf
index 8e97641..5c5b794 100644
--- a/mysql/files/my.cnf
+++ b/mysql/files/my.cnf
@@ -2,12 +2,6 @@
 # This config is generated by SALTSTACK
 # and all change will be overrided on next salt call
 {#-
-===== FETCH DATA =====
--#}
-{%- from "mysql/defaults.yaml" import rawmap with context -%}
-{%- from "mysql/supported_sections.yaml" import supported_sections with context -%}
-{%- set datamap = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) -%}
-{#-
 ===== COMBINE DATA =====
 -#}
 {%- set goodParamList = datamap.config.sections -%}
diff --git a/mysql/server.sls b/mysql/server.sls
index 1a28cc6..3624107 100644
--- a/mysql/server.sls
+++ b/mysql/server.sls
@@ -1,4 +1,5 @@
 {% from "mysql/defaults.yaml" import rawmap with context %}
+{% from "mysql/supported_sections.yaml" import supported_sections with context %}
 {%- set mysql = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:lookup')) %}

 {% set os = salt['grains.get']('os', None) %}
@@ -75,6 +76,9 @@ mysql_config:
     - name: {{ mysql.config.file }}
     - template: jinja
     - source: salt://mysql/files/my.cnf
+    - defaults:
+      supported_sections: supported_sections
+      datamap: {{ salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) }}
     - watch_in:
       - service: mysqld
     {% if os_family in ['Debian', 'Gentoo', 'RedHat'] %}
-- 
2.1.4

[gravyboat]

@ogabrielsantos This only had to be added for salt-ssh though right? Worked fine with the master/minion and masterless?

[ogabrielsantos]

@gravyboat Yep. Currently only salt-ssh needs the extra_filerefs. master/minion and masterless works fine without any intervention.

[gravyboat]

Hmm, this is very interesting. I don't really FEEL like you should have to do that. I'm going to leave this open for right now and comment in the other issue. Stealing your extra_filerefs content and will attribute to you.

[iggy]

There have been a couple issues in the past about extra_filerefs in a couple different places. It would be really nice if SaltStack could handle that on it's own, but until then, maybe we should start adding something to the README in all the formulas?

[gravyboat]

@iggy Yeah that would probably be a good plan until we see what happens with saltstack/salt#21370