ansible / content_collector

Build an Ansible collection and migrate content from an Ansible distribution using Ansible

Content Collector

The Ansible content collector is an Ansible playbook designed to assist in the creation of an Ansible collection and migrate content from an Ansible distribution into that collection.

For an overview of Ansible collections, see the Ansible documentation. https://docs.ansible.com/

Getting started

Clone this repo

git clone https://github.com/ansible/content_collector.git

The content collector requires an Ansible distribution from which content will be mirgated.

cd content_collector
wget https://releases.ansible.com/ansible/ansible-latest.tar.gz
tar -xvf ansible-latest.tar.gz

Update the site.yaml with the desired collection details:

The following are required:

• ansible_source_directory: The location of the Ansible distribution from which content will be migrated (e.g. ./ansible-2.8.2)
• destination_directory: The destination directory in which the collection will be built. (e.g. ~/my_ansible_collection)
• collection_namespace: The collection namespace. (e.g. contoso)
• collection_name: The name of the collection. (e.g. myos)
• source_sub_directory: The source directory within modules and module_utils from which content will be migrated. (e.g. /network/myos)

The following are optional and only used for the galaxy.yml file:

• collection_description: A description of the collection
• collection_version: The version of the collection (default=1.0.0)
• collection_readme: The readme file for the collection (default='README.md')
• collection_authors: A list of collection authors (default=[])
• collection_dependencies: A dictionary of collection dependencies (default={})
• collection_license: A list of licenses for the collection (default=[])
• collection_tags: A list of tags for the collection (default=[])
• collection_repository: The collection URL (default='')
• collection_documentation: The collection documentation URL (default='')
• collection_homepage: The colection homepage URL (default='')
• collection_issues: The collection issues URL (default='')

Details for the galaxy metadata fields can be found here: https://galaxy.ansible.com/docs/contributing/creating_collections.html#collection-metadata

Run the Ansible playbook:

ansible-playbook site.yaml

Update the galaxy.yml

A galaxy.yml file was generated in the root of the collection directory. Update as needed.

Details

1. Modules, module_utils, docs and test will be migrated into the collection.
2. Plugins will need to be manually migrated.
3. Testing is limited to ImportError detection by running each module.

Example resulting collection structure

my_ansible_collection
└── contoso
    └── myos
        ├── docs
        ├── playbooks
        ├── plugins
        │   ├── action
        │   ├── filter
        │   ├── inventory
        │   ├── modules
        │   │   ├── __init__.py
        │   │   ├── myos_module1.py
        │   │   ├── myos_module2.py
        │   │   └── myos_module3.py
        │   └── module_utils
        │       └── network
        │           └── myos
        │               ├── __init__.py
        │               └── myos.py
        └── roles

content_collector's Issues

README link for galaxy.yml metadata goes to wrong place

Currently it links to: https://galaxy.ansible.com/docs/contributing/creating_collections.html#collection-metadata

But the link should be updated to https://docs.ansible.com/ansible/devel/dev_guide/collections_galaxy_meta.html#collections-galaxy-meta

Support Ansible relative imports in a collection

ansible/ansible#59465 <-- when the code lands for this feature. Update content_collector to support relative imports in collections.

'Create a symbolic link' task too generic

When running this playbook, I got the error:

TASK [test_modules : Create a symbolic link] ***************************************************************************
fatal: [127.0.0.1]: FAILED! => changed=false 
  gid: 20
  group: staff
  mode: '0755'
  msg: refusing to convert from directory to symlink for /Users/jgeerling/.ansible/collections/ansible_collections/community
  owner: jgeerling
  path: /Users/jgeerling/.ansible/collections/ansible_collections/community
  size: 96
  state: directory
  uid: 501

The problem is, I already have other collections in that namespace that I do not want to have overwritten. It would be better to first make sure the namespace directory exists, then symlink the generated collection into that directory, instead of symlinking the entire namespace directory.

Use 'connection: local'?

I presume this would normally be run on a local workstation, and when I first tried running it, I got:

TASK [scaffold_directories : Create the collection directory structure] ************************************************
[WARNING]: Unhandled error in Python interpreter discovery for host 127.0.0.1: Failed to connect to the host via ssh:
ssh: connect to host 127.0.0.1 port 22: Connection refused

failed: [127.0.0.1] (item=~/.ansible/ansible-collections) => 
  ansible_loop_var: item
  item: ~/.ansible/ansible-collections
  msg: |-
    Data could not be sent to remote host "127.0.0.1". Make sure this host can be reached over ssh: ssh: connect to host 127.0.0.1 port 22: Connection refused
  unreachable: true

I had to add connection: local to the site.yaml playbook (or add -c local to the ansible-playbook command) to get it to work.

Extra double-nesting, galaxy.yml in wrong place

My testing is consistent with the file structure given in the example, but the example does not mention where galaxy.yml goes. I find that it is here:

my_ansible_collection
├── galaxy.yml
└── contoso
    └── myos
        ├── docs
        ├── playbooks
        ├── plugins
        │   ├── action
        │   ├── filter
        │   ├── inventory
        │   ├── modules
        │   │   ├── __init__.py
        │   │   ├── myos_module1.py
        │   │   ├── myos_module2.py
        │   │   └── myos_module3.py
        │   └── module_utils
        │       └── network
        │           └── myos
        │               ├── __init__.py
        │               └── myos.py
        └── roles

If I cd into my_ansible_collection, installing this does not seem to work.

It should not be in the folder my_ansible_collection, but instead in the myos, assuming that the ansible-galaxy collection install command is ran from the myos folder.

In my case, I'm using the tower modules, so I start by exporting it via this tool., site.yml:

- hosts: localhost
  gather_facts: false
  vars:
    ansible_source_directory: ./ansible
    destination_directory: ~/tower_modules
    collection_namespace: awx
    collection_name: awx
    source_sub_directory: /web_infrastructure/ansible_tower
  roles:
  - scaffold_directories
  - migrate_content
  - test_modules
  - galaxy_yml

Then I run ansible-playbook site.yaml

Example steps that install incorrectly

ansible-galaxy collection build --output-path=~/Documents/repos/utility-playbooks/test_build_bad
ansible-galaxy collection install ~/Documents/repos/utility-playbooks/test_build_bad/awx-awx-1.0.0.tar.gz -p ~/Documents/repos/utility-playbooks/test_install_bad
# (in utility playbooks)
ANSIBLE_COLLECTIONS_PATHS=$(pwd)/test_install_bad/ ansible-playbook -i localhost, -e ansible_python_interpreter=~/.virtualenvs/cli/bin/python tower_module.yml

This does not use the exported collection, but instead uses the modules as it already exists in Ansible.

Note location of modules, that's a little too many folders

test_install_bad/ansible_collections/awx/awx/awx/awx/plugins/modules/tower_organization.py

Example steps that install correctly

cd tower_modules/awx/awx/
cp ../../galaxy.yml galaxy.yml
ansible-galaxy collection build --output-path=~/Documents/repos/utility-playbooks/test_build
ansible-galaxy collection install ~/Documents/repos/utility-playbooks/test_build/awx-awx-1.0.0.tar.gz -p ~/Documents/repos/utility-playbooks/test_install
ANSIBLE_COLLECTIONS_PATHS=$(pwd)/test_install/ ansible-playbook -i localhost, -e ansible_python_interpreter=~/.virtualenvs/cli/bin/python tower_module.yml

Location of modules here

test_install/ansible_collections/awx/awx/plugins/modules/tower_organization.py

So my conclusion is that something is wrong with where this tool is putting things.

Doc fragments are not migrated

I left some details on how one can currently structure a collection with doc fragments here:

ansible/ansible#56226 (comment)

Right now, this tool will detect dependencies in Ansible module_utils. Dependencies in doc_fragments should also be automatically detectable, but it is not currently done.

Thus, in order to migrate a collection fully, a user needs to add doc_fragments folder in the plugins folder and copy over the files that used to live in the old folder by the same name.

site.yml vs. site.yaml

README says .yml but file is .yaml.