harisekhon / pylib

Python / Jython Utility Library for my other repos

Hari Sekhon - Python / Jython Library

git.io/python-lib

Python library, full of lots of validation code and utility functions.

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(you're welcome to connect with me on LinkedIn)

Build + Unit Tests

make &&
make test

Continuous Integration is run on this repo to build and test it (around 450 unit tests, plus custom tests).

Configuration

Strict validations include host/domain/FQDNs using TLDs which are populated from the official IANA list, a snapshot of which is shipped as part of this project.

To update the bundled official IANA TLD list with the latest valid TLDs do

make tld

Custom TLDs

If using bespoke internal domains such as .local, .intranet, .vm, .cloud etc. that aren't part of the official IANA TLD list then this is additionally supported via a custom configuration file resources/custom_tlds.txt containing one TLD per line, with support for # comment prefixes. Just add your bespoke internal TLD to the file and it will then pass the host/domain/fqdn validations.

Related Repositories

• Java version of this library
• Perl version of this library

More Related Repositories

• DevOps Bash Tools - 1000+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

• SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

• Jenkins - Advanced Jenkinsfile & Jenkins Groovy Shared Library

• GitHub-Actions - GitHub Actions master template & GitHub Actions Shared Workflows library

• Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

• Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

• Terraform - Terraform templates for AWS / GCP / Azure / GitHub management

• DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

• DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• Nagios Plugin Kafka - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support

• HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

• Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

• HashiCorp Packer templates - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init

• Diagrams-as-Code - Cloud & Open Source architecture diagrams with Python & D2 source code provided - automatically regenerated via GitHub Actions CI/CD - AWS, GCP, Kubernetes, Jenkins, ArgoCD, Traefik, Kong API Gateway, Nginx, Redis, PostgreSQL, Kafka, Spark, web farms, event processing...

git.io/python-lib

git.io/python-lib

pylib's Issues

Bug in warning threshold detection with the check_kafka.py plugin

I hit a strange bug with the check_kafka.py script when the kafka publish and consume takes longer than the warning threshold (1 second) it overrides the NagiosPlugin object's self.warning() method and sets it as an integer of 1. For now I just modified check_kafka.py to have much higher default thresholds.

The error shows up as an UNKNOWN response from the plugin with the message "UNKNOWN: 'int' object is not callable "

I added some debug print messages in check_kafka.py, then in pubsub_nagiosplugin.py, and finally in nagiosplugin.py. I think the bug is somewhere in the threshold checking mechanisms in nagiosplugin.py.

[root@ap-bornge02 pylib]# python /opt/nagios-plugins/pylib/check_kafka.py.debug --topic HealthCheckNagios
Returning
checking consumed message 'Test message from Hari Sekhon check_kafka.py.debug on host ap-bornge02.docl.nic at epoch 1479217115.2 (Tue Nov 15 08:38:35 2016) with random token 'ZD55usdANyDRfJ8LCEvv'' == published message 'Test message from Hari Sekhon check_kafka.py.debug on host ap-bornge02.docl.nic at epoch 1479217115.2 (Tue Nov 15 08:38:35 2016) with random token 'ZD55usdANyDRfJ8LCEvv''
checking self._consumed_message is not equal to self.publish_message....
setting self.message to my name or something
appending to self.msg with some time crap
running self.check_thresholds().....
Do I even have a val for self._publish_time? = 1.268
Calling self.get_threshold()...
checking self.warning() or self.critica().....
Calling self.get_threshold()...
checking self.warning() or self.critica().....
Warning in name so calling self.warning().....
type of self.warning = '1'
UNKNOWN: 'int' object is not callable

Initial Update

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create separate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖