uvsmtid / common-salt-states Goto Github PK

At the moment only win7 is reported as long as logic sees Windows.

This is the primary system overview of host role assignments and network assignments.

It is very useful to keep reviewing this table before large system deployment.

This only happens when plugins are initially installed (because plugin installation is not triggered if plugin is already available) because plugin installation triggers restart of Jenkins.

At the moment there is no proper detection of the moment when restart of Jenkins completes. Therefore, subsequent states are run immediately while Jenkins is still restarting and they fail.

There should be a way to detect when Jenkins restart completes (and wait until this time).

This is supposed to be a comprehensive configuration data which defines all repositories of any kind under system_repositories:

• filesystem_repositories
• svn_repositories
• git_repositories
• yum_repositories
• winrepo_repositories
• etc.

Instead of there should only be something like common.repositories_symlinks. The same applies to /srv/sources and /srv/resources which should become /srv/repositories with flat list of symlinks (without subdirectories).

It is not optimal to manage all types of repositories via single sub-key (e.g. differentiating them by repo_type field). A lot of them are drastically different (e.g. yum and git). Instead, it is better to have a sub-key per repository type (e.g. pillar['system_repositories']['git_repositories'][repon_name]). However, at the same time, absolute filesystem path is identical configuration for all of them.

Deliverables:

• All items in system_resources are supposed to refer to one of the repository under system_repositories.
• Rename depository into repository.
• Move configuration metadata from various system_features into respective keys under system_repositories.
• Stop using pillars/profile/common/system_features/resource_repositories_configuration.sls.
• Keep in mind #24 (generic publisher). Repositories should configure access to physical location, publishers should configure configuration how the repository is published. In particular, remove current common.source_symlinks and common.resource_symlinks - they simply become salt:// publisher configuration.
  How to differentiate host roles in this case? Not all repositories reside on the same machine. Does it mean there will be one role for each type for repositories? For example:
  • file_repository_role
  • svn_repository_role
  • git_repository_role
  • yum_repository_role
  • winrepo_repository_role
  • etc.
• Keep in mind #22 (listing absolute paths in properties).

Listing all absolute paths inside properties.yaml (and leaving only relative path configuration in the rest of pillars) can reduce differences between environments and make properties.yaml the focus of these differences in single location.

Currently, pillars are configured (by configure_salt.py script) on Salt master side as defaults and overrides:

pillar_roots:
    base:
    - /srv/pillars/overrides
    - /srv/pillars/defaults

• overrides is a symlink to pillars directory of separate repository for pillars;
• defaults is a symlink to pillars directory of project_name states repository.

However, it is common that most of the content in pillars directory of project_name states repository is duplicated. This requires constant cherry-picking of changes in pillars. Making pillars directory the bottom-level (the fallback, the first-level) defaults values would make this unnnecessary.

In case of initial-online-node bootstrap node, Salt master should only be installed on the node which was assigned controller_role role (or at least similar one).

At the moment controller_role is overused in multiple cases while it is still not clear what exactly "controller" means (except that it is a centralized node which controls possibly everything).

One of the separate roles is salt_master_role. If there is nothing else, then trivial renaming controller_role => salt_master_role makes purpose of the role clearer.

At the moment, gpg installation and configuration (besides just private key import) for Salt master has to be done manually - bootstrap package won't handle this.

Figure out how to include additional metadata to JSON schema to verify:

• Proper data structure in pillar (treat as error).
• Availability of pillar file in expected location (treat as warning).
• Availability of doc file in expected location (treat as warning).

Additional things:

• Figure out the process how to extend schema by adding project-specific data.

The implementation should probably be based on json schema.

Whether host is assigned address or not should be configured under system_networks by listing host_id under.

This splits hosts configuration and helps keep unchanged parts of it in isolation (in different files) highlighting true differences only during comparison.

There should be a decision made whether to keep MAC addresses (and list of interfaces) under host configuration or host assignment. It seems like the best idea is to use them in both places as the key which associates assignment (MAC belongs to network interface property under host configuration and MAC belongs to IP mapping under system_networks). Automated scripts can help to verify any inconsistencies.

There is simply a clone of the same pillars repository (labelled as bootstrap-target) which points to the branch with configuration required for selected target. The package is always generated for single target profile only.

In other words, if target is always single (and defined by currently checked out branch in bootstrap-target pillars repository), there is no need to list it.

This information (profile_name) is part of target profile pillars. So, the package can be generated without modifying any pillar files (be it source or target pillars).

Centralized identity management is common requirement.

More importantly, it allows "Create mutual trust with other Identity Management systems like Microsoft Active Directory".

At the moment only primary_user's password is configured through system_secrets.

There is also no clear procedure (and automatic support for it) how to plug in arbitrary storage (e.g. specific path on the filesystem with all secret data provided) so that Salt can use it. Adding key-values pairs through pillar has this bad encouragement that one day someone will commit the secret data into pillars repository.

Security

It is required to to provide secret pillar data to specific minions only, otherwise any minion can request anything via salt-call pillar.items.

UPDATE: It seems there is no issue to continue using Jinja templates in pillars (as long as no grains except id are used) because pillars are rendered on Salt master before being sent to minions - see this answer and this question.

This is often required to lay out the IP network across physical location.

Both production and development environments may benefit from it.

This is to be warned if any custom project still uses insecure (demo) keys and passwords.

At the moment, if minions are provisioned using Vagrant, the (virtual) network is not available until vagrant up is used. It is not necessarily linked to Vagrant, virtual network availability must be there regardless of the tool which creates it (Vagrant is just the most used method currently).

Many states which require communication to Salt master via this generic (virtual) network fail.

It also constitutes (partial) "chicken and egg" problem:

• In order to connect over this network, highstate has to be run (to get Vagrantfile configured to run vagrant up for minions).
• In order to complete highstate successfully, network should be already created (via Vagrant).

This is to support native file server for Windows minions.

At the moment a number of shared_* hosts are named without any naming convention.

Fix it by nameing them after the role they perform.
For example, shared_jenkins should be named after jenkins_master_role as shared_jenkins_master (without _role postfix).

And add the rest of shared_* hosts because not all roles are covered.

At the moment bootstrap script does not work on Windows.

There should be an easy way to make a conveniently setup Windows machine.

The idea is that many of the system_repositories (see #2) can be published on a web server.

As of now, there are different roles for that:

• depository_role
• vagrant_box_publisher_role
• local_yum_mirrors_role
• salt_master_role
• ...

Instead of specialized case-by-case publishers, there can be a generic configuration which tells:

• which repository (from system_repositories)
• which way to publish (HTTP, SMB, FTP, salt://, SSH, etc.)
• which role among defined ones is responsible for this configuration.

Consider both (double indirection):

• physical location of resources (configured under system_repositories - see see #2) on the filesystem
• logical location of resources exposed through specific URI_scheme

It is architecturally (conceptually) clear to have (e.g. webserver) configuration and layout the same for each repository server. However, physical storage requirements make it difficult (without logical redirection via mounting, symlinking, etc.) to place large content on fast (expensive) default filesystem location where webserver expects it to be (when filesystem root / and /var and /etc directories are not split across different mounts by default).

NOTE:
This is already implemented de facto in several roles:

• depository_role
• local_yum_mirrors_role
• vagrant_box_publisher_role
  They all have content directories (which should, in fact, be absolute paths configured per one of the system_repositories).

At the moment, Windows bootstrap script wrapper bootstrap.ps1 takes separate arguments for project_name, profile_name, host_id.

This complicates typing (as the file-based auto-completion is not available for separate parameters) and it is inconsistent with direct invocation of bootstrap.py (if it was on Linux).

Also, add two example scripts *.sh (for Linux) and *.cmd (for Windows) available inside bootstrap package as ready to be modified templates (instead of typing all arguments and output capture on the command line) - see deploy.md.

There is a step-by-step example how to set up private hosting for Vagrant boxes.

• Add additional vagrant hosting role which sets up an HTTP server to download boxes.
• If enabled (e.g. in properties), generated Vagrantfile should point URL to the host assigned to the vagrant hosting role.

If pillars still need to contain information about target profile name, path to directory with pillars, etc., it is more convenient to use command line parameters (i.g. parameter pillars for state.sls function) to make this data available because bootstrap package generation process is a single-shot execution which does not need to be constantly changed and committed in revisioned files.

In other words, it should be IMpossible to grep target profile name in pillar files belonging to source profile - this is simply not checked in.

The SSH keys deployed to primary user should not be under default name.Instead, use another name and configure ~/.ssh/config file to use these keys for all hosts within the system.

At the moment all primary users (and only primary users) have exactly the same insecure ssh keys for all hosts.

Ideally, there should be specification which host is connected with which default user and which ssh key. In most generic way, each username@hostname is configured with list of tuples (remote account, private key, public key), for example:

• alice@mercury: path/to/alice/mercury/{private,public}_key
• bob@jupiter: path/to/bob/jupiter/{private,public}_key

Currently, script generate_bootstrap_package.sh generates bootstrap for source environment.

It is actually not worth much - all it does is making it more convenient (not to type lengthy commands) to run bootstrap with overridden enable_bootstrap_target_envs and verify nothing has failed.

So, this script will be even more worthless because there will be no need to craft command line. Bootstrap generation is, in fact, supposed to generate package for single target environment only anyway - see #26. And enable_bootstrap_target_envs is supposed to be removed by #28.

To make more useful allow building bootstrap for any target profile name - checkout specified branch in target pillar repository (according to profile name specified on the command line).

In order to keep entire system description in pillars (to provide for "single setup logic - multiple deployment metadata" approach) firewall configuration should be placed into pillars.

The main difficulty is that configuration should basically support all options firewall can support.

Use only one of the three *bootstrap_target_envs keys:

• load_bootstrap_target_envs
• bootstrap_target_envs (preferred)
• enable_bootstrap_target_envs

There is no need for this anymore as load_bootstrap_target_envs does not need to be in Salt master configuration file.

They were needed when pillars were not able to be parameterized without master restart. Now it is possible through properties.yaml.

At the moment, the script only configures Salt master. It is not a issue most of the time as configuring Salt minion is nothing more than just setting its id.

However, this becomes more useful when it comes to master minion configuration. If Salt minion is not configured consistently (e.g. has different id), then setup skips running some initial states leaving host misconfigured. It takes some time to troubleshoot.

It becomes even more useful if this becomes common code among configure_salt.py and bootstrap.py. When offline-minion-installer bootstrap case is used, configuration of Salt minion is almost identical to Salt master (because all repositories are configured directly for Salt minion). Running the same code paths improves code coverage and consolidates code updates in single location.

Another case is clearing authentication keys and master public key to turn current minion into another.

There is no need to create symlinks within source profile pillars repository ("overrides") and all other repositories ("commons" and "defaults"). This can be achieved using filesystem paths overlays by configuring file_roots in Salt master (or masterless minion) via configure_salt.py script.

The benefits are:

• No noisy diff (or meld) output containing also target profiles during comparison.
• Cleaner repositories (no symlinks automatically generated by configure_salt.py).

Make sure to update populate_initial_project_repositories.sh which will require no bootstrap subdirectories in all pillars.

Salt configuration file (normally /etc/salt/master) in bootstrap package currently contains bootstrap configuration:

# ...
this_system_keys:
    # ...
    load_bootstrap_target_envs:
        automate-env: ~
# ...

However, necessary symlinks under pillars/bootstrap/profiles directory are not created during execution of bootstrap script:

DEBUG:root:command line: "salt-call" "--out" "json" "--log-level" "debug" "state.sls" "common.source_symlinks" "test=False" "pillar={ 'bootstrap_mode': 'initial-online-node' }"
DEBUG:root:stdout_file_path: /tmp/tmpvDa_hb
DEBUG:root:stdin_string: None
DEBUG:root:before wait
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Mako not available
[DEBUG   ] LazyLoaded .returner
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
DEBUG:root:after wait
{
    "local": [
        "Pillar failed to render with the following messages:", 
        "Specified SLS 'bootstrap.profiles.automate-env' in environment 'base' is not available on the salt master"
    ]
}

This is useful for system monitoring.

At the moment salt-minion-2014.7.1-1.el7.noarch already causes problems on CentOS 5 for common.tmux state which would otherwise execute cleanly on salt-minion-2015.5.5-1.el7.noarch.

----------
          ID: deploy_tmux_resurrect_plugin
    Function: archive.extracted
        Name: /opt
      Result: False
     Comment: 
     Started: 16:53:33.675349
    Duration: 9.463 ms
     Changes:   
              ----------
              pid:
                  6441
              retcode:
                  2
              stderr:
                  tar: This does not look like a tar archive
                  tar: --: Not found in archive
                  tar: -s: Not found in archive
                  tar: -t: Not found in archive
                  tar: -r: Not found in archive
                  tar: -i: Not found in archive
                  tar: -p: Not found in archive
                  tar: --: Not found in archive
                  tar: -c: Not found in archive
                  tar: -o: Not found in archive
                  tar: -m: Not found in archive
                  tar: -p: Not found in archive
                  tar: -o: Not found in archive
                  tar: -n: Not found in archive
                  tar: -e: Not found in archive
                  tar: -n: Not found in archive
                  tar: -t: Not found in archive
                  tar: -s: Not found in archive
                  tar: -=: Not found in archive
                  tar: -1: Not found in archive
                  tar: -f: Not found in archive
                  tar: '/var/cache/salt/minion/files/base/_opt_tmux-resurrect.tar': Not found in archive
                  tar: Exiting with failure status due to previous errors
              stdout:

At the moment, if system contains multiple minions, the resources are extracted as many times as there are minions (even though they are the same).

More over, there should be no generation of bootstrap for more than one environment.
Currently, a list of all possible environments pre-configured is maintained in pillars and automatically set symlinks by configure_salt.py script. This is wrong as this pre-configuration is inadequate (it is only partially true) because target profile pillars are in different branch of repository checked out into a single filesystem path and there is no automatic branch switching while bootstrap is being generated by bootstrap.generate_content Salt state. So, no matter how many target profile names are listed, the generated bootstrap packages become identical - is is just a waste of time.
This, however, is naturally addressed by issue #28 (there will be no way to specify target profile name except checking out required branch in the target pillars repository located in the single filesystem path) effectively making target profile name selection a single value.

Currently, entire Jenkins configuration is specified in a single file. This is probably the biggest file in all pillars. Split all jobs into separate pillar files to improve comparison via diff/meld/...

Introduce new top-level pillar key system_tasks (AKA jobs) which could potentially be generalized for any type of jobs (Jenkins, Cron, Salt, etc.). There is no generalization within the scope of this task, only re-factoring.

So, similarly to system_repositories where each type of repository is listed under sub key (like yum, git, etc.), Jenkins jobs are supposed to be listed under system_tasks:jenkins_tasks.

Keep in mind:

• Ideally, all tasks should be thought of as scripts (instantiations of templates) on the filesystem. In other words, each task should be runnable from command line. For example, Jenkins job simply runs required script (in addition to being configured in Jenkins itself).

Otherwise, it leads to inconsistency between what target profile pillars data was used in bootstrap package generation and what got exported from repositories into bootstrap package. If these are different HEADs, the bootstrap package is not the one expected.

• 1. Stop using both load_bootstrap_target_envs and enable_bootstrap_target_envs - see #13.
• 2. Stop listing target profile names for bootstrap package generation - see #28.
• 3. Export current HEADs (or current branches) instead of branch names specified in pillars - see #29.
• 4. Use command line parameters to dynamically provide values for bootstrap package generation - see #27.
• 5. Optimize time required for bootstrap package generation - see #26.
• 6. Use filesystem paths overlays in file_roots to provide target pillar data (instead of symlinks in source repositories) - see #30.

This is an alternative to configuration of hosts files.