fiunchinho / ansible-aws-elasticsearch-module Goto Github PK
View Code? Open in Web Editor NEWAnsible module to create a new AWS ElasticSearch cluster
Ansible module to create a new AWS ElasticSearch cluster
I am using this custom Elasticsearch Service with Ansible. The following Config produces ValidationException
.
- hosts: 127.0.0.1
connection: local
tasks:
- name: "Create ElasticSearch cluster"
ec2_elasticsearch:
name: "Testing"
elasticsearch_version: "2.3"
region: "us-east-1"
instance_type: "t2.medium.elasticsearch"
instance_count: 1
dedicated_master: False
zone_awareness: False
ebs: True
volume_type: "standard"
volume_size: 10
snapshot_hour: 1
access_policies: "{{ lookup('file', 'es_policies.json') | from_json }}"
register: es_info
Output with -vvv
No config file found; using defaults
PLAYBOOK: setup_elasticsearch.yaml *********************************************
1 plays in data_pipeline/ansible/setup_elasticsearch.yaml
PLAY [127.0.0.1] ***************************************************************
TASK [setup] *******************************************************************
Using module file /Users/nehiljian/.pyenv/versions/2.7.11/envs/recommender_env/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nehiljain
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922 `" && echo ansible-tmp-1486674667.03-193453987758922="` echo $HOME/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/gn/zdn4js5n7m96lpdtvx7h7d440000gn/T/tmpfDSuhx TO /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/ /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/nehiljian/.pyenv/versions/2.7.11/envs/recommender_env/bin/python2.7 /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/setup.py; rm -rf "/Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/" > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1]
TASK [Create ElasticSearch cluster] ********************************************
task path: ansible/setup_elasticsearch.yaml:6
Using module file ansible/library/ec2_elasticsearch.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nehiljain
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884 `" && echo ansible-tmp-1486674668.31-219225481684884="` echo $HOME/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/gn/zdn4js5n7m96lpdtvx7h7d440000gn/T/tmpfajB91 TO /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ec2_elasticsearch.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ec2_elasticsearch.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/nehiljian/.pyenv/versions/2.7.11/envs/recommender_env/bin/python2.7 /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ec2_elasticsearch.py; rm -rf "/Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/" > /dev/null 2>&1 && sleep 0'
fatal: [127.0.0.1]: FAILED! => {
"changed": false,
"failed": true,
"invocation": {
"module_args": {
"access_policies": {
"Statement": [
{
"Action": "es:*",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXX:user/APIAccess"
},
"Resource": "arn:aws:es:us-east-1:XXXXXXXXXX:domain/xxxxx/*"
},
{
"Action": "es:*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"xx.xx.xx.xx"
]
}
},
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Resource": "arn:aws:es:us-east-1:XXXXXXXXXX:domain/xxxxx/*",
"Sid": ""
}
],
"Version": "2012-10-17"
},
"aws_access_key": null,
"aws_secret_key": null,
"dedicated_master": false,
"dedicated_master_instance_count": null,
"dedicated_master_instance_type": null,
"ebs": true,
"ec2_url": null,
"elasticsearch_version": "2.3",
"instance_count": 1,
"instance_type": "t2.medium.elasticsearch",
"name": "Testing",
"profile": "default",
"region": "us-east-1",
"security_token": null,
"snapshot_hour": 1,
"validate_certs": true,
"volume_size": 10,
"volume_type": "standard",
"zone_awareness": false
},
"module_name": "ec2_elasticsearch"
},
"msg": "Error: ValidationException"
}
to retry, use: --limit @ansible/setup_elasticsearch.retry
PLAY RECAP *********************************************************************
127.0.0.1 : ok=1 changed=0 unreachable=0 failed=1
Any clue what is wrong? Thanks
I'm using echo -e "{{ec2_access_key}}\n{{ec2_secret_key}}\n{{aws_region}}\njson\n" | aws --profile aws.sandbox configure
to create the profile. Should I be using a different method?
ansible 2.2.1.0
aws-cli/1.11.49 Python/2.7.13 Darwin/16.4.0 botocore/1.5.12
main.yml
---
- name: "Create ElasticSearch cluster"
ec2_elasticsearch:
name: "elastic-cloud"
elasticsearch_version: "5.1"
region: "{{aws_region}}"
instance_type: "m3.medium.elasticsearch"
instance_count: 1
dedicated_master: False
#dedicated_master_instance_type: "t2.micro.elasticsearch"
#dedicated_master_instance_count: 2
zone_awareness: False
ebs: False
volume_type: "standard"
volume_size: 10
snapshot_hour: 0
access_policies: "{{ lookup('file', 'access_policy.json') | from_json }}"
profile: "aws.sandbox"
register: response
AWS IAM Profile Policy:
AmazonRDSFullAccess
AmazonEC2FullAccess
IAMFullAccess
AmazonElastiCacheFullAccess
AmazonS3FullAccess
AWSElasticBeanstalkFullAccess
AmazonVPCFullAccess
AmazonRoute53FullAccess
Hello,
I was looking at using this earlier but noticed a license mismatch: the meta/main.yml file says MIT, but the ec2_elasticsearch.py says GPL3. Which is it?
Perhaps something with the aws sdk changed but I wasn't able to get this module to work for me.
Traceback (most recent call last):
File "/tmp/ansible_xGoIah/ansible_module_ec2_elasticsearch.py", line 239, in <module>
main()
File "/tmp/ansible_xGoIah/ansible_module_ec2_elasticsearch.py", line 225, in main
AccessPolicies=pdoc,
File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/client.py", line 228, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/client.py", line 468, in _make_api_call
api_params, operation_model, context=request_context)
File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/client.py", line 521, in _convert_to_request_dict
api_params, operation_model)
File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/validate.py", line 269, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "ElasticsearchVersion", must be one of: DomainName, ElasticsearchClusterConfig, EBSOptions, AccessPolicies, SnapshotOptions, AdvancedOptions
Here is my config:
- name: "Create ElasticSearch cluster"
ec2_elasticsearch:
name: "my-cluster-test"
elasticsearch_version: "2.3"
region: "us-east-1"
instance_type: "t2.small.elasticsearch"
instance_count: "3"
zone_awareness: False
dedicated_master: False
ebs: True
volume_type: "gp2"
volume_size: 50
snapshot_hour: 13
access_policies: "{{ lookup('template', playbook_dir + '/roles/retraced/templates/elasticsearch_iam.yml.j2') }}"
profile: "default"
register: response
thanks!
I ran into errors with this module when setting ebs and dedicated_master to false.
I suggest something like the following-
esconfig = {
'InstanceType': module.params.get('instance_type'),
'InstanceCount': int(module.params.get('instance_count')),
'DedicatedMasterEnabled': module.params.get('dedicated_master'),
'ZoneAwarenessEnabled': module.params.get('zone_awareness')
}
ebsoptions = {
'EBSEnabled': module.params.get('ebs')
}
if esconfig['DedicatedMasterEnabled']:
esconfig['DedicatedMasterType'] = module.params.get('dedicated_master_instance_type')
esconfig['DedicatedMasterCount'] = int(module.params.get('dedicated_master_instance_count'))
if ebsoptions['EBSEnabled']:
ebsoptions['VolumeType'] = module.params.get('volume_type')
ebsoptions['VolumeSize'] = int(module.params.get('volume_size'))
response = client.create_elasticsearch_domain(
DomainName = module.params.get('name'),
ElasticsearchClusterConfig = esconfig,
EBSOptions = ebsoptions,
SnapshotOptions = {
'AutomatedSnapshotStartHour': module.params.get('snapshot_hour')
}
)
My fork is currently the only published version at this stage, be good to have the original up.
If you run this module twice, it will put the cluster into a processing state, even if there are no changes. Since the processing state takes 10-20 minutes to finish, this is a big time sink.
There should probably be a request to describe the elasticsearch domain and only perform updates if there are differences in configuration?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.