netdevops / hier_config Goto Github PK
View Code? Open in Web Editor NEWHierarchical Configuration
License: MIT License
Hierarchical Configuration
License: MIT License
We need a fast/simple method of loading a deep series of children e.g.
deepest_child = config.add_children_deep([
'router ospf 123',
'area 0',
'interface bundle-e1',
'cost 100'])
deepest_child.text
> 'cost 100'
router ospf 123
area 0
interface bundle-e1
cost 100
This would be handy for setting up tests.
I am adding functionality for HPE Comware switches. However, as VLAN configuration mixes multiple vlans into the same configuration statement things are getting a bit complicated. I have written a pre/post processor for dealing with it. However from architectural standpoints this is a mess.
Could you please advise on a more generic approach to implement it?
See my request for comments branch on this: https://github.com/empi89/hier_config/tree/rfc-pre-and-postprocessing
Python 3.7
On Windows 10
Fresh git Fork and Clone of netdevops / hier_config
When I run the Unit Tests I get one failure
Testing started at 22:19 ...
C:\Users\Rich\.virtualenvs\hier_config-UnGuqua-\Scripts\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2018.2.2\helpers\pycharm\_jb_unittest_runner.py" --path C:/Users/Rich/PycharmProjects/hier_config/tests
Launching unittests with arguments python -m unittest discover -s C:/Users/Rich/PycharmProjects/hier_config/tests -t C:\Users\Rich\PycharmProjects\hier_config\tests in C:\Users\Rich\PycharmProjects\hier_config\tests
C:\Users\Rich\.virtualenvs\hier_config-UnGuqua-\lib\site-packages\pep8.py:110: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
Error
Traceback (most recent call last):
File "C:\Program Files (x86)\Python37-32\Lib\unittest\case.py", line 59, in testPartExecutor
yield
File "C:\Program Files (x86)\Python37-32\Lib\unittest\case.py", line 615, in run
testMethod()
File "C:\Users\Rich\PycharmProjects\hier_config\tests\test_hier_config.py", line 69, in test_load_from_file
hier.load_from_file(myfile.name)
File "C:\Users\Rich\PycharmProjects\hier_config\hier_config\hconfig.py", line 141, in load_from_file
with open(file_path) as f:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Rich\\AppData\\Local\\Temp\\tmpv_hmd0_7'
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:32: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
cls.hconfig_tags = yaml.load(open(cls.tags_file))
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:32: ResourceWarning: unclosed file <_io.TextIOWrapper name='C:\\Users\\Rich\\PycharmProjects\\hier_config\\tests\\files\\test_tags_ios.yml' mode='r' encoding='cp1252'>
cls.hconfig_tags = yaml.load(open(cls.tags_file))
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:33: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
cls.hconfig_options = yaml.load(open(cls.options_file))
C:\Users\Rich\PycharmProjects\hier_config\tests\test_host.py:33: ResourceWarning: unclosed file <_io.TextIOWrapper name='C:\\Users\\Rich\\PycharmProjects\\hier_config\\tests\\files\\test_options_ios.yml' mode='r' encoding='cp1252'>
cls.hconfig_options = yaml.load(open(cls.options_file))
Ran 60 tests in 0.073s
FAILED (errors=1)
Process finished with exit code 1
The temp File is created and the contents are written to it, I am unable to open it manually, The security permissions show as full access for my user. If I copy the file, I can open the copy and inspect the contents. It appears to be a lock issue, unique to Windows. It is discussed here:
https://bugs.python.org/issue14243
The Fix would be to close the file before calling load_from_file, then manually clean up
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:38
Tagging rules can be written in such a way as to produce stranded children in the remediation config for a particular tag. While this is infrequent, it can be painful to track down.
class HConfigChild(object):
@property
def tags(self):
my_tags = set(self._tags)
for child in self.children:
my_tags.update(child.tags)
return mytags
add_tags
, deep_append_tags
, etc. as neededExaggerated example
- lineage
- startswith: interface
add_tag: interfaces_with_children
- lineage
- startswith: interface
- anything: null
remove_tags: interfaces_with_children
Remediation config with interfaces_with_children tag:
interface fa0/1
interface fa0/2
Does anyone have knowledge of others extending this project to Juniper or Nokia or other devices?
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:205
CodeFactor found an issue: The if statement can be replaced with 'return bool(test)'
It's currently on:
hier_config\hc_child.py:47
client 10.1.1.1 server-key
client 10.2.2.2 server-key
#e.g.
idempotent_commands:
- lineage:
- re_search: ^client {client_ip} server-key
idempotent_commands:
- lineage:
- startswith: hardware access-list tcam region
- lineage:
- startswith: hardware profile tcam region
If you were to lump them together, an idempotent command check for hardware profile tcam region X
would be True if hardware access-list tcam region Y
were in the compiled_config. Much of the time, this is still okay but there are cases where it is not. In certain cases, this behavior can shadow command negations.
lineage
over and overidempotent_commands:
- - startswith: aaa authentication attempts
- - startswith: aaa authentication login
- - startswith: aaa authentication enable
- - startswith: interface # Support for deep rules
- startswith: description
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:129
Maybe I'm missing something but the banner check will not work on Nexus devices. The format of my banner is this:
banner motd #
Multiline
text
here.
#
No matter how I run it or what tweaks I make, it has this result:
File "/home//lib64/python3.8/site-packages/hier_config/root.py", line 402, in _load_from_string_lines
assert not in_banner, "we are still in a banner for some reason"
AssertionError: we are still in a banner for some reason
Obviously it thinks it's still in the banner and not detecting the "#" as the end of the banner. I've tried on multiple Nexus devices with the same result. It works fine on IOS devices. I can provide more information if needed. If there's a set work around for this then maybe I've missed it.
More info:
hier-config==2.2.0
I'm also making sure I'm setting the style to "nxos".
Thanks!
Currently, we have the capability to compare configurations with the goal of building remediation commands to get from one configuration state to another. It would be useful to provide a diff functionality that understands the hierarchy of network device configurations in a way that the diff command does not.
+ ntp server 192.168.1.1
- ntp server 192.168.1.2
interface fa0/1
+ ip address 172.16.1.1 255.255.255.0
- ip address 172.16.2.1 255.255.255.0
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:814
In reference to #11, our team has been discussing Python 2.7 support, in addition to the Python 3.6 support that the code is currently written to. We certainly appreciate the contributions, but go back and forth on whether it makes sense to walk back code to support 2.7.
Our main concerns with supporting 2.7 are:
Soon to be deprecated.
Ability to take advantage of Python 3.x enhancements:
f
and r
strings make core more efficient and easier to read.Given this, we'd like to be able to support the community. We'd like to propose some options:
hier_config
would likely be converted to Python 3 only. However, Python 2.7 versions would remain on pypi and the last release supporting Python 2.7 would be preserved in a python2
branch.hier_config
to specifically support Python 2.7.CodeFactor found an issue: Redefining built-in 'file'
It's currently on:
hier_config\host.py:100
Commit f44aedb
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:47
EOS uses 3 spaces instead of 2 for indenting children.
CodeFactor found an issue: Trailing spaces
It's currently on:
.travis.yml:25
rnanney [2:58 PM]
thanks for putting this on github. just fyi I used pip to install but had trouble instantiating Host. I had to add the following to __init__.py:
`from hier_config.host import Host`
CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().
It's currently on:
tests\test_hier_config.py:38
Python 3.8.13 (default, Mar 17 2022, 00:00:00)
hier-config==2.0.1
nornir-hier-config==0.1.0
I would like to generate a remediation file for a Cisco Nexus device. I found that some lines seem to be omited when you choose nxos os.
Create two files:
running-config.txt:
tacacs-server deadtime 3
tacacs-server host 192.168.1.99 key 7 "Test12345"
template.txt:
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3
the script when i set nxos
from a host.
from hier_config.host import Host
from hier_config import HConfig
host = Host(hostname='test', os='nxos')
running = HConfig(host=host)
running.load_from_file('running-config.txt')
compiled = HConfig(host=host)
compiled.load_from_file('template.txt')
rem = running.config_to_get_to(compiled)
for line in rem.all_children():
print(line.cisco_style_text())
no tacacs-server deadtime 3
no tacacs-server host 192.168.1.99 key 7 "Test12345"
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3
When I set host = Host(hostname='test', os='nxos')
to host = Host(hostname='test', os='ios')
no tacacs-server deadtime 3
no tacacs-server host 192.168.1.99 key 7 "Test12345"
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3
no tacacs-server deadtime 3
no tacacs-server host 192.168.1.99 key 7 "Test12345"
tacacs-server host 192.168.1.98 key 0 Test135 timeout 3
tacacs-server host 192.168.100.98 key 0 test135 timeout 3
Add support for IOS-XR configuration comments
router isis backbone
set-overload-bit on-startup 180
is-type level-2-only
! ISIS network number should be encoded with 0-padded loopback IP
! example:
! example:
net 49.0001.1921.2022.0222.00
I'm thinking the comments should be a property of the configuration line just below the comment lines.
How to remove comments:
https://supportforums.cisco.com/t5/xr-os-and-platforms/cisco-ios-xr-remove-comment/td-p/2546154
If PyYaml is required, then this project should not support Python 3.4
https://pyyaml.org/wiki/PyYAML states:
Requirements
PyYAML requires Python 2.7 or Python 3.5+.
hier_config has the ability to load many instances of remediation into a single object tree. This provides the basis for creating reports over large sets of hier_config remediation configs e.g.:
line vty 0 4
are there?Create documentation for this feature set.
There is currently no documentation besides some comments in the files them selves. We need to expand in the docs.
As a network automation engineer, I'd appreciate the ability to modify the remediation configuration lines.
This is useful for situations where a negation line needs to be in a certain format for the network OS to accept it. For example, removing an SNMP user on Cisco Nexus requires the line to be truncated after the username and will error if the full line is presented.
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:764
Test full configuration workflows in a circular way that ensures correctness of hier_config's remediation and future config generating features.
running.future(remediation)
future.future(rollback)
While generating the remediation commands, also generate rollback configuration. The rollback should be tied to the remediation in such a way as to receive the same tags. Rollback config can be generated today by reversing the commands_to_get_to()
operation.
Running config
hostname router1.dfw
interface fa1
ip address 192.168.1.1 255.255.255.0
Compiled config
hostname router1-new.dfw
interface fa1
ip address 192.168.2.1 255.255.255.0
interface fa2
ip address 192.168.3.1 255.255.255.0
Remediation config
hostname router1-new.dfw
interface fa1
ip address 192.168.2.1 255.255.255.0
interface fa2
ip address 192.168.3.1 255.255.255.0
Rollback config
hostname router1.dfw
interface fa1
ip address 192.168.1.1 255.255.255.0
no interface fa2
Since going from hier-config 1.4.0 to 1.4.1 and using the very latest hconfig-remediation-ansible hier-config breaks with the following error:
ansible-playbook -i inventory test.pb.yml -v
Using /etc/ansible/ansible.cfg as config file
PLAY [all] *******************************************************************************************************************************
TASK [BUILD REMEDIATION CONFIG WITHOUT TAGS] *********************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: load_tags() got an unexpected keyword argument 'file'
fatal: [router1]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_3zgHIj/ansible_module_hconfig_remediation.py\", line 189, in <module>\n main()\n File \"/tmp/ansible_3zgHIj/ansible_module_hconfig_remediation.py\", line 167, in main\n host.load_tags(hier_tags, file=False)\nTypeError: load_tags() got an unexpected keyword argument 'file'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
to retry, use: --limit @/storage1/ansible-hconfig/hconfig-remediation-ansible-master/example/test.pb.retry
PLAY RECAP *******************************************************************************************************************************
router1 : ok=0 changed=0 unreachable=0 failed=1
Downgrading back to hier-config 1.4.0 everything works again.
Looks like the codefactor_fixes broke things
I do not get a match under ordering
when I use a nested lineage (interface subcommands); but it works when I only match the parent line.
Here is my config data:
Running:
interface GigabitEthernet1/0/18
ip access-group TEST in
!
ip access-list extended TEST
permit tcp host 10.1.1.1 host 10.1.1.2 eq 443
!
end
Intended:
interface GigabitEthernet1/0/18
ip access-group TESTING in
!
ip access-list extended TESTING
permit tcp host 10.1.1.2 host 10.1.1.1 eq 443
!
end
Here is a non-working ordering
Options:
ordering:
- lineage:
- startswith:
- ip access-list
- access-list
order: 100
- lineage:
- startswith: interface
- startswith: ip access-group
order: 200
- lineage:
- startswith:
- no ip access-list
- no access-list
order: 300
idempotent_commands:
- lineage:
- startswith: interface
- startswith:
- ip access-group
Result:
"ip access-list extended TESTING",
" 10 permit tcp host 10.1.1.2 host 10.1.1.1 eq 443",
"no ip access-list extended TEST", ## This should have a weight of 300 (it actually does based on teh next results) ##
"interface GigabitEthernet1/0/18", ## The next two lines should have a weight of 200 (does not since the above is 300) ##
" ip access-group TESTING in"
I tested without using a nested lineage and I get the expected results
Options:
ordering:
- lineage:
- startswith:
- ip access-list
- access-list
order: 100
- lineage:
- startswith: interface ## Only matching the parent line ##
order: 200
- lineage:
- startswith:
- no ip access-list
- no access-list
order: 300
idempotent_commands:
- lineage:
- startswith: interface
- startswith:
- ip access-group
Result:
"ip access-list extended TESTING",
" 10 permit tcp host 10.1.1.2 host 10.1.1.1 eq 443",
"interface GigabitEthernet1/0/18",
" ip access-group TESTING in"
"no ip access-list extended TEST",
ref #14
Exclude tags appears to omit some remediation steps. In the output below the commands:
interface Vlan3
description switch_mgmt_10.0.3.0/24
interface Vlan4
description switch_mgmt_10.0.4.0/24
should be present, but the interface
parent commands are missing from the description
commands.
>>> from hier_config import Host
>>> host = Host(hostname="aggr-example.rtr", os="ios")
>>> host.load_tags_from_file("tags_ios.yml")
>>> host.load_running_config_from_file("running_config.conf")
>>> host.load_generated_config_from_file("generated_config.conf")
>>> print(host.remediation_config_filtered_text(include_tags=[], exclude_tags=["manual"]))
vlan 3
name switch_mgmt_10.0.3.0/24
vlan 4
name switch_mgmt_10.0.4.0/24
description switch_mgmt_10.0.3.0/24
description switch_mgmt_10.0.4.0/24
When the below empty banner is defined, _load_from_string_lines()
does not recognize the end of the banner resulting in AssertionError: we are still in a banner for some reason
.
banner motd ##
I'm trying to ensure that an ACL gets created before any other commands. It seems that by lowering the order ensures that it will get created before everything else which has a value of 500. (As a test I've also changed it so it's a greater number than 500 and still makes no difference)
I've set the below in my options.yml file but it doesn't take affect, the output file always shows that the ACL is going to be applied in the same place.
ordering:
- lineage:
- startswith:
- ip access-list
order: 1
CodeFactor found an issue: Redefining built-in 'file'
It's currently on:
hier_config\host.py:181
Commit f44aedb
Create remediation fix up functions for IOS/NXOS/EOS/IOS-XR that safely remove unused objects such as:
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:762
CodeFactor found an issue: Unnecessary "else" after "return"
It's currently on:
hier_config\hc_child.py:198
CodeFactor found an issue: The if statement can be replaced with 'return bool(test)'
It's currently on:
hier_config\hc_child.py:814
Load default options, that can be extended upon with the hconfig_options
.
CodeFactor found an issue: init method from base class 'HConfigChild' is not called
It's currently on:
hier_config_init_.py:52
CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().
It's currently on:
hier_config\host.py:225
Commit f44aedb
CodeFactor found an issue: Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().
It's currently on:
tests\test_hier_config.py:41
As a network automation engineer, I want the minimum required package versions to be bumped to the latest versions to resolve conflicts with other automation platform packages.
$ mypy --strict hier_config/
hier_config/options.py:1: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:16: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:79: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:183: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:320: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:558: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:689: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/options.py:691: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/base.py:67: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/base.py:151: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/base.py:191: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:40: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:41: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:91: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:350: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:418: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:418: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/child.py:419: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/child.py:446: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/child.py:469: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/child.py:481: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/root.py:83: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/root.py:147: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/root.py:175: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/root.py:196: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/host.py:44: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/host.py:44: error: Incompatible default for argument "hconfig_options" (default has type "None", argument has type "Dict[Any, Any]")
hier_config/host.py:51: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/host.py:107: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
hier_config/host.py:140: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/host.py:168: error: Implicit generic "Any". Use "typing.List" and specify generic parameters
hier_config/host.py:168: error: Implicit generic "Any". Use "typing.Dict" and specify generic parameters
Found 32 errors in 5 files (checked 7 source files)
CodeFactor found an issue: The if statement can be replaced with 'return bool(test)'
It's currently on:
hier_config\hc_child.py:129
Create a dynamic remediation that makes working with access-lists hitless.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.