chaffelson / nipyapi

A convenient Python wrapper for Apache NiFi

NiPyApi

Nifi-Python-Api: A rich Apache NiFi Python Client SDK

Features

Three layers of Python support for working with Apache NiFi:

• High-level Demos and example scripts
• Mid-level Client SDK for typical complex tasks
• Low-level Client SDKs for the full API implementation of NiFi and selected sub-projects

Functionality Highlights:

• Detailed documentation of the full SDK at all levels
• CRUD wrappers for common task areas like Processor Groups, Processors, Templates, Registry Clients, Registry Buckets, Registry Flows, etc.
• Convenience functions for inventory tasks, such as recursively retrieving the entire canvas, or a flat list of all Process Groups
• Support for scheduling and purging flows, controller services, and connections
• Support for fetching and updating Variable Registries
• Support for import/export of Versioned Flows from NiFi-Registry
• Docker Compose configurations for testing and deployment
• A scripted deployment of an interactive environment, and a secured configuration, for testing and demonstration purposes

Please see the issue register for more information on current development.

Quick Start

There are several scripts to produce demo environments in nipyapi.demo.*
The mid-level functionality is in nipyapi.canvas / nipyapi.security / nipyapi.templates / nipyapi.versioning
You can access the entire API using the low-level SDKs in nipyapi.nifi / nipyapi.registry

The easiest way to install NiPyApi is with pip:

# in bash
pip install nipyapi

You can set the config for your endpoints in the central config file:

# in python
import nipyapi
nipyapi.config.nifi_config.host = 'http://localhost:8080/nifi-api'
nipyapi.config.registry_config.host = 'http://localhost:18080/nifi-registry-api'

Then import a module and execute tasks:

nipyapi.canvas.get_root_pg_id()
>'4d5dcf9a-015e-1000-097e-e505ed0f7fd2'

You can use the Docker demos to create a secured interactive console showing many features:

from nipyapi.demo.secured_console import *
from nipyapi.demo.console import *

You can also explore the scripts to get ideas for how NiPyAPi can be used to automate your environment.

Please check out the Contribution Guide if you are interested in contributing to the feature set.

Background and Documentation

For more information on Apache NiFi, please visit https://nifi.apache.org
For Documentation on this package please visit https://nipyapi.readthedocs.io.

NiFi Version Support

Currently we are testing against NiFi versions 1.1.2 - 1.23.2, and NiFi-Registry versions 0.1.0 - 1.23.2.
If you find a version compatibility problem please raise an issue

Python Support

Python 2.7 or 3.7-12 supported, though other versions may work.
We will shortly stop supporting Python2.
OSX M1 chips have various issues with Requests and Certificates.

Tested on AL2023, developed on OSX 14.2 - Windows testing not attempted.
Outside of the standard Python modules, we make use of lxml, DeepDiff, ruamel.yaml and xmltodict in processing, and Docker for demo/tests.

nipyapi's Issues

Support Import from URI/File/HTTP/etc.

It would be handy to allow users to import a Flow directly from something other than files and objects already in memory, suggestions so far include:

• Import from a Gist
• Import from a URI

Error listing remote process group

• Nipyapi version: 0.12.0
• NiFi version: 1.1.1
• NiFi-Registry version: None
• Python version: 3.6.6
• Operating System: Ubuntu 16.04.4 LTS

Description

Hi

I have a workflow which has a process group called 'transfer learning' under the 'root' process group. The 'transfer learning' process group has a remote process group named 't'. I'm trying to list all remote process groups under the 'transfer learning' process group using:

nipyapi.canvas.list_all_remote_process_groups(<id of the 'transfer learning' process group>)

But I got an empty list.

When I use the same command WITHOUT the ID of the 'transfer learning' process group (i.e., using the default 'root' process group), then I got the correct list of remote process groups

I think there is a problem in the implementation of the 'list_all_by_kind' function as it list all process groups under the specified pg_id first, then searches for the required 'kind' under each process group. I think the implementation should initially search for the required 'kind' under the specified 'pg_id', then searching for the required 'kind' under each child process group if 'descendants=True'

Urgency

Not very urgent, as I used the 'nipyapi.nifi.ProcessGroupsApi().get_remote_process_groups' to list the required remote process groups

Regards

How to handle insecure certificates?

Haven't found an option how to access nifi if it has insecure certificate?

Feature: Purge queues with a process group

From this StackOverflow question: https://stackoverflow.com/q/48710410/4717963
It would be useful to have convenience methods to purge all the flowfiles from a PG, particularly when conducting testing or if you wanted to force delete the PG.

Can't install in Alpine environments

• Nipyapi version: 0.8.0
• NiFi version: N/A
• NiFi-Registry version: N/A
• Python version: 3.6.5
• Operating System: (Alpine, specifically Docker image python:3-alpine)

Description

Describe what you were trying to get done, or what you would like the package to do.
Tell us what happened, what went wrong, and what you expected to happen.

Was attempting to use the library in a Docker image and opted originally for the Alpine variant. Confirmed that Debian based variants of python image seem to be okay

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

Sample Dockerfile to recreate

FROM python:3-alpine
RUN python --version
RUN pip install nipyapi

Building results in:

Collecting cffi>=1.7 (from cryptography>=1.3.4; extra == "security"->requests[security]->nipyapi)
Downloading https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz (438kB)
Complete output from command python setup.py egg_info:

    No working compiler found, or bogus compiler options passed to
    the compiler from Python's standard "distutils" module.  See
    the error messages above.  Likely, the problem is not related
    to CFFI but generic to the setup.py of any Python package that
    tries to compile C code.  (Hints: on OS/X 10.8, for errors about
    -mno-fused-madd see http://stackoverflow.com/questions/22313407/
    Otherwise, see https://wiki.python.org/moin/CompLangPython or
    the IRC channel #python on irc.freenode.net.)

Add secure NiFi and secure Registry to test environment

User, group, and access policy functions cannot be tested by the test suite without a secure NiFi and secure Registry. The NiFi Docker image supports TLS certificate-based authentication and LDAP authentication.

Enhance registry deployment functionality

Get_registry_bucket

• Add exact vs greedy match to allow for name reuse

Change set_endpoint to use hard logout

Add verify_ssl option to login/endpoint options for https

Deployment checks (Sensitive, Invalid, etc.)

• Add option to filter by PG
• Add option to output summary info required for remediation (ID, Properties)
• Look for Mandatory but Unset properties

Cannot create new user group populated with users

• Nipyapi version: 0.12.0
• NiFi version: 1.8.0
• NiFi-Registry version:
• Python version: 3.7.2
• Operating System: OS X

Description

nipyapi.nifi.TenantsApi().create_user_group() is unable to create a group populated with users, because the UserEntity contains read-only properties, which the NiFi API rejects as a part of an incoming change.

What I Did

nifi_users = nipyapi.security.list_service_users()

OR

nifi_users = nipyapi.nifi.TenantsApi().get_users()

...

nipyapi.nifi.UserGroupEntity(revision = nipyapi.nifi.RevisionDTO(version = 0), component = 
nipyapi.nifi.UserGroupDTO(identity = group, users = nifi_users.users))

Error:

Traceback (most recent call last):
  File "./init_nifi/nifi_create_users_and_groups.py", line 76, in <module>
    group_obj = nipyapi.nifi.TenantsApi().create_user_group(group_create_obj)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/apis/tenants_api.py", line 172, in create_user_group
    (data) = self.create_user_group_with_http_info(body, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/apis/tenants_api.py", line 253, in create_user_group_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/api_client.py", line 326, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/api_client.py", line 371, in request
    body=body)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/rest.py", line 268, in POST
    body=body)
  File "/usr/local/lib/python3.7/site-packages/nipyapi/nifi/rest.py", line 224, in request
    raise ApiException(http_resp=r)
nipyapi.nifi.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 16 Jan 2019 20:23:17 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'Content-Type': 'text/plain', 'Vary': 'Accept-Encoding', 'Content-Length': '607', 'Server': 'Jetty(9.4.11.v20180605)'})
HTTP response body: Unrecognized field "userGroups" (class org.apache.nifi.web.api.dto.TenantDTO), not marked as ignorable (6 known properties: "parentGroupId", "versionedComponentId", "position", "id", "identity", "configurable"])
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 414] (through reference chain: org.apache.nifi.web.api.entity.UserGroupEntity["component"]->org.apache.nifi.web.api.dto.UserGroupDTO["users"]->java.util.HashSet[0]->org.apache.nifi.web.api.entity.TenantEntity["component"]->org.apache.nifi.web.api.dto.TenantDTO["userGroups"])

Urgency

Unable to continue development using nipyapi without additional custom code

UX: Bootstrap an empty NiFi instance from a given flow definition

As a deployer, I want to be able to publish to a registry and have a live NiFi cluster deploy a given flow version.

This is not merely a 'deploy the flow in NiFi workflow'. The focus is on the initial setup for a post-install script. I.e. given a running NiFi cluster and registry, issue a command to deploy a registry controller service in NiFi and ultimately have it deploy a given flow (a la startup flow). This will involve populating the registry with this startup flow as an intermediary step.

Registry api_client is null for imports

• Nipyapi version: nipyapi 0.12.0
• NiFi version: 1.5 HDF 3.1.2
• NiFi-Registry version: 0.10 HDF 3.1.2
• Python version: 2.7.5
• Operating System: RedHat 7 Latest

Description

Describe what you were trying to get done, or what you would like the package to do.
Tell us what happened, what went wrong, and what you expected to happen.

Since I'm stuck on a version of NiFi Registry Prior to Git Support and Proper Import Export Capability I'm putting together some command line tools. I'm able to export and run other registry commands just fine but on the import it's failing because nipyapi.config.registry_config.api_client is None

Traceback (most recent call last):
File "./import_flow", line 7, in
flow = nipyapi.versioning.import_flow_version(sys.argv[1], flow_id=sys.argv[2], file_path=sys.argv[3])
File "/home/1454256952/.local/lib/python2.7/site-packages/nipyapi/versioning.py", line 685, in import_flow_version
dto=dto
File "/home/1454256952/.local/lib/python2.7/site-packages/nipyapi/utils.py", line 101, in load
return nipyapi.config.registry_config.api_client.deserialize(
AttributeError: 'NoneType' object has no attribute 'deserialize'

Here is how I'm setting up the config

import nipyapi
import subprocess

nifiToken = subprocess.check_output(['curl','-s','-k','-X POST','--negotiate','-u :','https://example.org:9091/nifi-api/access/kerberos'])
registryToken = subprocess.check_output(['curl','-s','-k','-X POST','--negotiate','-u :','https://example.org:61443/nifi-registry-api/access/token/kerberos'])

#nipyapi.config.nifi_config.debug=True
#nipyapi.config.registry_config.debug=True

nipyapi.config.nifi_config.host = 'https://example.org:9091/nifi-api'
nipyapi.config.registry_config.host = 'https://example.org:61443/nifi-registry-api'
nipyapi.security.set_service_ssl_context(service='nifi', ca_file='/home/1454256952/.pki/trust.pem', client_cert_file=None, client_key_file=None, client_key_password=None)
nipyapi.security.set_service_ssl_context(service='registry', ca_file='/home/1454256952/.pki/trust.pem', client_cert_file=None, client_key_file=None, client_key_password=None)
nipyapi.security.set_service_auth_token(token=nifiToken, token_name='tokenAuth', service='nifi')
nipyapi.security.set_service_auth_token(token=registryToken, token_name='tokenAuth', service='registry')

#!/bin/python

import config
import nipyapi
import sys

flow = nipyapi.versioning.import_flow_version(sys.argv[1], flow_id=sys.argv[2], file_path=sys.argv[3])

What I Did

To get past this I updated utils.py to actual set nipyapi.config.registry_config.api_client = ApiClient() right before the call to deserialize but I'm not sure how nipyapi.config.registry_config.api_client is supposed to get set.

Urgency

I've got a workaround so not too urgent just not sure of the implications.

Add support for NiFi-1.6.0

NiFi-1.6.0 has been released, NiPyApi should aim to stay current

UX: List and diff available flow versions locally and across Registry instances

As a deployer, I want to cross-check available versions in 2 or more instances of the registry to understand if one has a newer version of the flow.

Question - what's the best way to idempotently identify a flow version across registry instances. Maybe we should request the registry enhancement to have a fingerprint of the version in its metadata? I.e. given registries A and B, confirm that Av2 is the same as e.g. Bv5.

Feature idea for nipyapi

This is a very rich python library for managing NiFi. While reading through it I found that we can retrieve, instantiate, upload, and export templates. However is it also possible to do the following?

multiple template management orchestration
The idea is to administer multiple templates on the canvas by storing their state programmatically. The reason we need to store it's state programmatically is that NiFi REST API doesn't support following template functions.
1:) start template by template instance name
2:) stop template by template instance name

Also, On a different note, is it possible to do the following.
1:) clear the canvas
2:) clear/empty the queues of the connections by connection ID

I can contribute in case you agree with the idea and it's not implemented yet.

Report underlying HTTP requests for executed commands

Description - Feature Proposal

I am proposing a new feature for NiPyAPI. Many times, when a user wants to know what REST API calls to execute to achieve a specific goal, we instruct them to use the UI and their browser's Developer Tools panel to monitor which API calls the UI client makes to the server. NiPyAPI exposes a number of these tasks, as well as providing convenience methods that recurse through process groups/components, etc. to make these operations even easier. It would be nice if the user could perform one or more operations via NiPyAPI and then have the series of API calls exported via a number of possible outputs.

Terminology

• operation - a single action executed by NiPyAPI. This could be one or more HTTP API requests to NiFi
• API request - an HTTP request made by the NiPyAPI client to the NiFi server
• curl - the curl command-line tool & library
• session - an individual interaction with the NiPyAPI command-line tool, which can contain 0 or more operations. Terminated when the client process is killed

"Ken Burns Mode" could be enabled in a variety of ways:

• Invoke NiPyAPI with a flag like --echoHttp so all operation(s) executed during a NiPyAPI session get echoed, in sequence, to a specific output
  • console output
  • could be piped to a separate output file
• When running in a NiPyAPI session, a specific operation could be invoked with a flag like --echoHttp to print the API request(s) associated with the single operation
  • A specific operation could be invoked with a flag like --dry-run in conjunction with the --echoHttp flag to print the API request that would be sent without actually sending it
• A curl compatibility mode could be enabled in any/all of the above scenarios to print the HTTP API request(s) in a form that could be copy/pasted to the command line and invoked via curl (i.e. headers are broken out into -H "X-Header-Name: headerValue" form)

Examples

• A normal example is getting the application status, which is a single operation in NiPyAPI and a single API call
• A complex example is getting the processors with sensitive properties, which I believe requires iterating over multiple components via an API call for each

Secured NiFi cluster support

Does the package support secured NiFi clusters?

Not able to install on py 3.7

• Python version: 3.7
• Operating System: Windows 10 Pro

Description

I am not able to install nipyapi using pip. I check the pip list and it is showing latest packages.

What I Did

I ran pip install nipyapi and it gave me below error.

Complete output from command python setup.py egg_info:
Warning: 'keywords' should be a list, got type 'NoneType'
Traceback (most recent call last):
File "", line 1, in
File "C:\Users...\ruamel.yaml\setup.py", line 858, in
main()
File "C:\Users...\ruamel.yaml\setup.py", line 847, in main
setup(**kw)
File "c:\python\lib\site-packages\setuptools_init_.py", line 140, in setup
return distutils.core.setup(**attrs)
File "c:\python\lib\distutils\core.py", line 108, in setup
setup_distribution = dist = klass(attrs)
File "c:\python\lib\site-packages\setuptools\dist.py", line 370, in init
k: v for k, v in attrs.items()
File "c:\python\lib\distutils\dist.py", line 267, in init
getattr(self.metadata, "set" + key)(val)
File "c:\python\lib\distutils\dist.py", line 1203, in set_keywords
self.keywords = _ensure_list(value, 'keywords')
File "c:\python\lib\distutils\dist.py", line 40, in _ensure_list
value = list(value)
TypeError: 'NoneType' object is not iterable

Command "python setup.py egg_info" failed with error code 1 in C:\Users...\AppData\Local\Temp\pip-install-wjym60pp\ruamel.yaml\

NiFi-1.8.0 and NiFi-Registry-0.3.0 Support

NiPyApi should support the latest versions of the relevant clients

Secure Connection Example Request

• Nipyapi version: 0.9.1
• NiFi version: 1.6.0/1.7.0
• NiFi-Registry version: N/A
• Python version: 3.6.3
• Operating System: CentOS 7.x

Description

We are currently trying to learn/understand the authentication and authorization as pertains to NiFi. We have accomplished a basic setup using the Docker apache/nifi image and the tls-toolkit for generating the certificates/keystores. Now we are trying to figure out how to connect using Nipyapi. We have gone through the documentation but are just getting more confused as the requirements for the SSL context dont align with the what was generated with tls-toolkit output.

Personal, i barely understand SSL and every time we have to deal with it hurts my brain. Does anyone have a basic example of connecting to a secure NiFi standalone instance?

Note: we did go through "secure_connection.py" but it does not clear up our misunderstandings.

What I Did

The command we used with the tls-toolkit:

tls-toolkit.sh standalone -n localhost -C "CN=admin,OU=NIFI"

Urgency

We are currently in development, nothing is in production today. We started to use nipyapi to automate some initial deployment tasks. We were previously working with a unsecured instance, but are not attempting make use of NiFi's S2S capabilities across sites; as such we are trying to secure communications.

NiPyApi with Apache Knox gateway secured Nifi cluster.

• Nipyapi version: v0.11.0
• NiFi version: 1.7.1
• NiFi-Registry version:
• Python version: 3.6.2
• Operating System:

Description

NiPyApi with Apache Knox gateway secured Nifi cluster.

What I Did

I have a cluster with Apache Knox federation. I do not know how to use the NiPyApi project with such a setting.

Making REST calls against /nifi-api (secure) is pretty simple.

1. Make a GET on /knoxtoken/api/v1/token endpoint with user/password auth credentials, to receive a access_token
2. Use the access_token as a 'hadoop_jwt' cookie on subsequent REST calls against the secure /nifi-api endpoint

Does the NiPyApi project support this?

When I try to use nipyapi.security.service_login with auth creds, the nipyapi.nifi.configuration.dict['api_key'] is auto filled with the login page html.

or, If i use the access_token from Step 1, and patch it as the 'api_key' after making the service_login, on any subsequent usage (say nipayapi.canvas.list_process_groups(pg_id='root')) , I receive a 'Authorization Denied' message.

Any advise on how to get this to work?

Urgency

Not an urgent issue. Just curious on how to get this to work against a Knox gateway cluster

create connections between processors

From an email:
I tried something like

print nipyapi.nifi.ProcessGroupsApi().create_connection(id=pg_id,
body=nipyapi.nifi.models.connection_entity.ConnectionEntity(
source_group_id=pg_id,
source_type="PROCESSOR",
source_id=in_http.id,
destination_group_id=pg_id,
destination_type="PROCESSOR",
destination_id=out_kafka.id
)
)

but always got "Connection details must be specified"

could you please provide me with working example?

Add Hortonworks Schema Registry Support

• Nipyapi version: 0.80
• NiFi version: 1.5
• NiFi-Registry version: 0.1
• Python version: 2.7, 3.4
• Operating System: Linux

Description

Add support to read and write all major apis from Schema Registry. Especially to upload and download schemas. The API is full swaggerized and documented.

https://github.com/hortonworks/registry
http://registry-project.readthedocs.io/en/latest/
http://registry-project.readthedocs.io/en/latest/schema-registry.html#api-examples

XMLSyntaxError when trying to export template

• Nipyapi version: 0.7.0
• NiFi version: 1.1.0.2.1.2.0-10
• NiFi-Registry version:
• Python version: 3.6.4
• Operating System: Windows 8 x64

Description

I wanted to export a template I have created.
When calling the method templates.export_template, I receive an XMLSyntaxError.

What I Did

>>> from nipyapi import nifi, templates, config
>>> config.nifi_config.host = <nifihost>
>>> access_token = nifi.AccessApi().create_access_token(username=<snip>,password=<snap>)
>>> config.nifi_config.api_client = nifi.ApiClient(header_name='Authorization',header_value='Bearer {}'.format(access_token))
>>>
>>> temp = templates.get_template_by_name('test_template')
>>> r = templates.export_template(temp.id)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Projects\nifi-deploy\venv\lib\site-packages\nipyapi\templates.py", line 174, in export_template
    template_xml = etree.fromstring(response.data)
  File "src\lxml\etree.pyx", line 3230, in lxml.etree.fromstring (src\lxml\etree.c:81056)
  File "src\lxml\parser.pxi", line 1871, in lxml.etree._parseMemoryDocument (src\lxml\etree.c:121236)
  File "src\lxml\parser.pxi", line 1759, in lxml.etree._parseDoc (src\lxml\etree.c:119912)
  File "src\lxml\parser.pxi", line 1125, in lxml.etree._BaseParser._parseDoc (src\lxml\etree.c:114159)
  File "src\lxml\parser.pxi", line 598, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\etree.c:107724)
  File "src\lxml\parser.pxi", line 709, in lxml.etree._handleParseResult (src\lxml\etree.c:109433)
  File "src\lxml\parser.pxi", line 638, in lxml.etree._raiseParseError (src\lxml\etree.c:108287)
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1

Add certifi to dependency requirements

This project appears to require the certifi package. For now, pip install certifi is a fine workaround.

Result when importing nipyapi:

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:01:18) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import nipyapi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\__init__.py", line 17, in <module>
    importlib.import_module('nipyapi.' + sub_module)
  File "C:\TDE\Bin\Python\Python35-32\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\config.py", line 12, in <module>
    from nipyapi.nifi import configuration as nifi_config
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\nifi\__init__.py", line 285, in <module>
    from .apis.access_api import AccessApi
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\nifi\apis\__init__.py", line 4, in <module>
    from .access_api import AccessApi
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\nifi\apis\access_api.py", line 24, in <module>
    from ..api_client import ApiClient
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\nifi\api_client.py", line 29, in <module>
    from .rest import ApiException, RESTClientObject
  File "C:\TDE\Bin\Python\Python35-32\lib\site-packages\nipyapi\nifi\rest.py", line 19, in <module>
    import certifi
ImportError: No module named 'certifi'

Create add_user_group_to_access_policy

There is no high-level function for adding a group to an access policy.

Support Start/Stop calls for Process Groups and Processors

It would be useful for testing if deployed processors/process groups could be started and stopped.
This is particularly important to test congruence between import/export of Flows and Templates.

UX: Upload/Download a bucket/item/version of the flow from the registry

As a deployer I would like to issue a command to download a specific (or latest if version omitted) flow representation and save this artifact in a file. The file is to be in a format understood by another instance of the registry for import.

Bug - NiPyAPI 0.10 - get_flow_version method

• Nipyapi version: 0.10.0
• NiFi version:1.6
• NiFi-Registry version:0.2.0
• Python version:2.7
• Operating System: Redhat Linux

Description

I am trying to deploy the nifi template and found one issue in the below method, the version param is being asserted as stringtypes but it should be int types.

def get_flow_version(bucket_id, flow_id, version=None, export=False):
assert version is None or isinstance(version, six.string_types)

What I Did

deploy_flow_version(rootPg, location, bucketProdID, workflowProdID, registryID,ver)

Traceback (most recent call last):
File "", line 1, in
File "/home/nifi/v_env/nifi/lib/python2.7/site-packages/nipyapi/versioning.py", line 733, in deploy_flow_version
version=version
File "/home/nifi/v_env/nifi/lib/python2.7/site-packages/nipyapi/versioning.py", line 567, in get_flow_version
assert version is None or isinstance(version, six.string_types)
AssertionError

Urgency

ASAP

Can't Lookup Existing UserGroup

• Nipyapi version: 0.12.0
• NiFi version: 1.7
• NiFi-Registry version: N/A
• Python version: 3.6
• Operating System: CentOS 7.x

Description

I've successfully been able to modify the various policies in at the global and component level for a specific user. I am having trouble figuring out how to accomplish the same task for a group of users. The 'nipyapi.security.add_user_group_to_access_policy' function exists, but I do not see any way of obtaining/searching for a UserGroupEntity.

To be more specific my instance is setup with LDAP Authentication and when manually adding users I can select a group, which is oddly represented as a 'User'. When using 'nipyapi.security.list_service_users' groups do not show up as users, they only appear as components under user context in 'user_groups'.

There is the

What I Did

#Gets root processor ID
root_pg_id = nipyapi.canvas.get_root_pg_id()
#Gets user entity object if exists
user = nipyapi.security.get_service_user('myuser')
#Gets the current policy for the root processor group
policy = nipyapi.security.get_access_policy_for_resource('data', 'read', r_id=root_pg_id)
#Adds the user entity to the policy
nipyapi.security.add_user_to_access_policy(user, policy)

Urgency

non-urgent.

TypeError using get_access_policy_for_resource

• Nipyapi version: 0.9.1
• NiFi version: 1.6.0/1.7.0
• NiFi-Registry version: N/A
• Python version: 3.6.3
• Operating System: CentOS 7.4.1708

Description

In our current docker build/setup we are experiencing some issues with proper policies rights being set. We sought to use nipyapi to correct that. While learning/understanding how to create/modify/delete policy information in NiFi we attempted to use the nipyapi function "nipyapi.security.get_access_policy_for_resource" to view existing information. However using the call, regardless of the variations in options, fails to complete, errors on "TypeError: Got an unexpected keyword argument 'id' to method get_access_policy_for_resource".

What I Did

The test script:

nipyapi.nifi.configuration.verify_ssl = False
nipyapi.config.nifi_config.host = 'https://nifi:9443/nifi-api'
nipyapi.security.set_service_ssl_context(
    service='nifi',
    ca_file='nifi-cert.pem',
    client_cert_file='client-cert.pem',
    client_key_file='client-key.key',
    client_key_password='supersecretpassword'
)
canvas_root_policy = nipyapi.security.get_access_policy_for_resource('flow', 'read')
print(canvas_root_policy)

Fails with the following output:

Traceback (most recent call last):
  File "./test.py", line 37, in <module>
    canvas_root_policy = nipyapi.security.get_access_policy_for_resource('flow', 'read')
  File "/opt/cgimss-utils/lib/python3.6/site-packages/nipyapi/security.py", line 367, in get_access_policy_for_resource
    id=r_id
  File "/opt/cgimss-utils/lib/python3.6/site-packages/nipyapi/nifi/apis/policies_api.py", line 279, in get_access_policy_for_resource
    (data) = self.get_access_policy_for_resource_with_http_info(action, resource, **kwargs)
  File "/opt/cgimss-utils/lib/python3.6/site-packages/nipyapi/nifi/apis/policies_api.py", line 314, in get_access_policy_for_resource_with_http_info
    " to method get_access_policy_for_resource" % key
TypeError: Got an unexpected keyword argument 'id' to method get_access_policy_for_resource

When lookin into the code a bit we can see the call being made to "nipyapi.nifi.PoliciesApi().get_access_policy_for_resource" contains the keyword "id". Following it down to where the error comes from it appears it is just because that key is not listed in "all_params" on line 303 of "policies_api.py". Adding it correct's the behavior, though may not be the right way to fix the issue. It might also appear in other similar calls?

nipyapi/security.py

    360     try:
    361         if service == 'nifi':
    362             log.info("Getting NiFi policy for %s:%s/%s",
    363                      action, resource, r_id)
    364             return nipyapi.nifi.PoliciesApi().get_access_policy_for_resource(
    365                 action=action,
    366                 resource=resource,
    367                 id=r_id
    368             )

nipyapi/nifi/apis/policies_api.py

    303         all_params = ['action', 'resource']
    304         all_params.append('callback')
    305         all_params.append('_return_http_data_only')
    306         all_params.append('_preload_content')
    307         all_params.append('_request_timeout')
    308
    309         params = locals()
    310         for key, val in iteritems(params['kwargs']):
    311             if key not in all_params:
    312                 raise TypeError(
    313                     "Got an unexpected keyword argument '%s'"
    314                     " to method get_access_policy_for_resource" % key
    315                 )
    316             params[key] = val
    317         del params['kwargs']

Urgency

Currently in development, nothing is in production today. Started to use nipyapi to automate some initial deployment tasks. We encountering issues with Docker NiFi and policies, hoping to fix them with nipyapi. There are manual work arounds for the time being.

start_docker_containers always pulls even if an image exists locally

• Nipyapi version: 0.10.3
• NiFi version: 1.7.1
• NiFi-Registry version: 0.2.0
• Python version: 2.7.14 (Anaconda)
• Operating System: OSX 10.13.6

Description

start_docker_containers always pulls an image even if it exists locally.
This is inefficient (e.g. apache/nifi:1.7.1 is 1.72GB) and painful on slow connections.

What I Did

nipyapi.utils.start_docker_containers(
    d_containers,
    d_network_name
)

Urgency

Not urgent, have patched utils.py locally to try d_client.images.get(image) before falling back to pulling.
Will create a branch and send a pull request.

Rework procedurally generated docs into a consistent format before 1.0 release

• Nipyapi version: 0.3.2 and older
• Python version: Any
• Operating System: Any

Description

The Autogenerated documentation of the API from the Swagger Definition has resulted in a lot of duplication and bad formatting. It would further benefit from being broken up into navigable sections when browsed on ReadTheDocs

nipyapi.security.get_access_policy_for_resource does not work

• Nipyapi version: 0.12.0, master
• NiFi version: 1.8.0
• NiFi-Registry version: 0.3.0
• Python version: 3.7.2
• Operating System: OS X 10.12.6

Description

Calling nipyapi.security.get_access_policy_for_resource(action=<action>, resource=<resource>) returns None. Looking deeper, the r_id is never appended to the resource. Looking deeper than that, if you append resource + '/' + r_id, nipyapi.nifi.PoliciesApi().get_access_policy_for_resource attempts to escape the slash, and the API returns HTTP response body: The request was rejected because the URL contained a potentially malicious String "%2F".

Traceback (most recent call last):
  File "./initialize_nifi.py", line 296, in <module>
    action=action,
  File "/media/code/nipyapi/nipyapi/nifi/apis/policies_api.py", line 279, in get_access_policy_for_resource
    (data) = self.get_access_policy_for_resource_with_http_info(action, resource, **kwargs)
  File "/media/code/nipyapi/nipyapi/nifi/apis/policies_api.py", line 368, in get_access_policy_for_resource_with_http_info
    collection_formats=collection_formats)
  File "/media/code/nipyapi/nipyapi/nifi/api_client.py", line 326, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/media/code/nipyapi/nipyapi/nifi/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/media/code/nipyapi/nipyapi/nifi/api_client.py", line 349, in request
    headers=headers)
  File "/media/code/nipyapi/nipyapi/nifi/rest.py", line 233, in GET
    query_params=query_params)
  File "/media/code/nipyapi/nipyapi/nifi/rest.py", line 224, in request
    raise ApiException(http_resp=r)
nipyapi.nifi.rest.ApiException: (500)
Reason: Server Error
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 07 Feb 2019 21:20:22 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'Content-Length': '87', 'Server': 'Jetty(9.4.11.v20180605)'})
HTTP response body: The request was rejected because the URL contained a potentially malicious String "%2F"

I haven't tested this, but I suspect this may be due to the following code in __call_api:

    # path parameters
    if path_params:
        path_params = self.sanitize_for_serialization(path_params)
        path_params = self.parameters_to_tuples(path_params,
                                                collection_formats)
        for k, v in path_params:
            # specified safe chars, encode everything
            resource_path = resource_path.replace(
                '{%s}' % k, quote(str(v), safe=config.safe_chars_for_path_param))

Urgency

Moderate - continued development is hindered by needing to find a workaround to this issue.

Prevent commands unsupported for NiFi Version

As a user, it would be helpful if the module checked the version of NiFi it was connected to, and threw an error if I tried to run a command that was not tested as supported for that version.

This is particularly relevant for all the new version control commands, which won't work on older versions of NiFi anyway, so this would provide a tidy way to manage this.
It would also provide a convenient approach to controlling the matrix of which functions and versions are combined in regression testing.

Support enabling all controller services when dependency exists in them

We use nipyapi to enable controller services instead of clicking them in the UI, and very often one controller service depends on the other controller service(s), which means some controller services must be enabled before the other controller services.

One function I implement

def enable_controller_services_recur(controller_services):
    if not controller_services:
        return
    else:
        invalid_services = []
        for service in controller_services:
            if service.component.validation_status == 'VALID':
                nipyapi.canvas.schedule_controller(service, True)
            else:
                invalid_service = nipyapi.canvas.get_controller(identifier=service.id, identifier_type="id")
                invalid_services.append(invalid_service)
        enable_controller_services_recur(invalid_services)

and the input parameter I use is the processor group

controller_services = nipyapi.nifi.FlowApi().get_controller_services_from_group(pg.id,
                                                                                    include_descendant_groups=True)\
        .controller_services

I could send a merge request with a more generic function as the following if the maintainer think it's useful.

Proposal

I would put this function in canvas.py closed to the function schedule_controller(controller, scheduled), and make it behave like

def schedule_controllers(controllers, scheduled):
"""
    Start/Enable or Stop/Disable Controller Services with existing dependencies 

    Args:
        controllers list(ControllerServiceEntity): Target Controllers to schedule
        scheduled (bool): True to start, False to stop

    Returns:
        list(ControllerServiceEntity)
""

deploy_flow_version behaves differently in NiPy to GUI

• Nipyapi version: 0.11.0
• NiFi version: 1.7.1
• NiFi-Registry version: 0.3.0
• Python version: 3.7
• Operating System: Centos7

Description

When using the command deploy_flow_version you may receive an error like:

File "/home/nifi/v_env/nifi/lib/python2.7/site-packages/nipyapi/versioning.py", line 761, in deploy_flow_version
raise ValueError(e.body)
ValueError: Node XXXXX is unable to fulfill this request due to: No applicable policies could be found. Contact the system administrator. Contact the system administrator.

My guess is that this is caused by the NiFi GUI solely using the NiFI API and related AuthZ, whereas NiPy also calls the Registry API to ensure that the bucket/flow being targeted exists. This should be investigated and resolved, preferably by more closely matching the NiFi workflow.

Urgency

This will block users who have different permissions on their Registry implementation to their NiFi implementation.

pytest is not listed in project requirements

• Nipyapi version:0.12.0
• Python version:3.6.7

Description

I want to run the test, and find pytest is not listed in requirements.txt

What I Did

install pytest in my virtual env. However then package containspytest and it's not listed in project requirement

Add feature to deploy a flow version to a new Process Group

Overlooked in the 0.8 release was a wrapper function to deploy a flow from Version Control as a new Process Group, which is basically the last step of most code promotion processes.

Installation with python setup.py install does not work (without pip and no internet access)

• Nipyapi version:'0.9.1'
• NiFi version:NA
• NiFi-Registry version:NA
• Python version:2.7.5
• Operating System:RedHat RHEL 7.3 x86_64

Description

I don't have acces to an internet connection.
I do not use pip either.

When I tried to run the nipyapi won't install :
python setup.py install

Traceback (most recent call last):
File "setup.py", line 45, in
exclude=['.tests', '.tests.', 'tests.', 'tests']
TypeError: find_packages() got an unexpected keyword argument 'include'

Is there a way I can install without pip ?

What I Did

python setup.py install

Traceback (most recent call last):
File "setup.py", line 45, in
exclude=['.tests', '.tests.', 'tests.', 'tests']
TypeError: find_packages() got an unexpected keyword argument 'include'

Export nested versioned process-groups from Registry

• Nipyapi version: 0.10.0
• NiFi version: 1.7.1
• NiFi-Registry version: 0.3.0-SNAPSHOT
• Python version: 3ish
• Operating System: All

Description

When you import a PG into Nifi from a registry, you get the whole PG, and all nested; versioned PGs within it. I would have expected some way of achieving the same result from either the API, the toolkit, or nipyapi when doing an Export.

What I Did

If I export/get a flow via nipyapi - I only seem to get the top level flow, and just a pointer to the nested versioned PGs. Would be great if there was an option to recurse down the stack and get a fully working flow (in our case - for use in Minifi)

Urgency

Not super urgent - but would be nice to know what we can do.

Unable to connect to multiple NiFi's

• Nipyapi version: 0.10.3
• NiFi version: 1.7.1
• NiFi-Registry version: N/A
• Python version: 2.7.15
• Operating System: OSX 10.14

Description

I want to connect to different NiFi servers within the same script. I currently have a loop, but when looking at the data returned it is data from the first set server and not the other server. I have tried reloading the module each time. It would probably be easier to initialize a class instead of configuring a module. I would prefer to do something like the following:

import nipyapi


consumer_nifi = nipyapi.Server(host='consumer.nifi.int.example.com:80/nifi-api')
s3_nifi = nipyapi.Server(host='s3.nifi.int.example.com:80/nifi-api')

print(consumer_nifi.system.get_system_diagnostics())
print(s3_nifi.system.get_system_diagnostics())

What I Did

import nipyapi
import os

from pprint import pprint


nifi_clusters = ['consumer', 'cluster', 's3']
status_metrics = [
    'flow_files_in',
    'flow_files_out',
    'flow_files_queued',
    'flow_files_received',
    'flow_files_sent',
    'flow_files_transferred',
]


def clean_nifi_name(name):
    """Lowercase and remove spaces in NiFi names."""
    return name.lower().replace(' ', '_').replace('_-_', '_')


def lambda_handler(event, context):
    """Lambda handler."""
    env = os.environ['ENVIRONMENT']

    metrics_list = []

    for cluster in nifi_clusters:
        print("Getting Metrics for {}".format(cluster))
        nipyapi.config.nifi_config.host = '{}.nifi.int.example.com:80/nifi-api'.format(cluster)

        flows = nipyapi.canvas.recurse_flow().to_dict()['process_group_flow']['flow']
        process_groups = flows['process_groups']

        for process_group in process_groups:
            name = clean_nifi_name(process_group['component']['name'])

            print(name)

            for metric in status_metrics:
                point = process_group['status']['aggregate_snapshot'][metric]
                metrics_list.append({
                    'metric': "nifi.process_group.{}".format(metric),
                    'points': point,
                    'tags': [
                        "process_group:{}".format(name)
                    ],
                    'type': 'gauge'
                })

    pprint(metrics_list)


if __name__ == "__main__":
    os.environ["AWS_REGION"] = "us-west-1"
    lambda_handler(event={}, context={})

Urgency

We are unable to use this library because we need to connect to multiple NiFi server/clusters from within the same script.

UX: Publish a flow definition to the registry

As a deployer, I want to publish a given flow version to the registry. The command-line expectation is to provide a local file (or, better, URI, where file://whatever is just one instance) to be published. Additionally, one would provide mnemonic flow path in the registry and the system must resolve the correct bucket/item uuids.

Bonus: support 'create parent' buckets/items if not available via a simple cmd-line switch. E.g. think of the mkdir -p /path/which/may/not/yet/exist

Attach an existing process group to a version

• Nipyapi version: master
• NiFi version: 1.8.0
• NiFi-Registry version: 0.30

Hi - Is there any way to attach a process group to an existing version from version control?

I've trying to integrate building a flow as part of a CI process, pushing the newly built flow to a registry bucket on every commit.

I'm getting the error Error creating snapshot: A Versioned flow snapshot with the same version already exists: 1 on successive builds I presume is because the flow is being 'built' separately and then I'm trying to use save_flow_ver.

For example I have the following bit of code:-

try:
    group = "Extract"
    process_group = nipyapi.canvas.get_process_group(group)

    flow = nipyapi.versioning.get_flow_in_bucket(
        bucket_id=bucket.identifier, identifier=group
    )

    if flow is not None:

        # somehow tell this it's an existing flow
        # let the new flow think it's an update without checking it out first
        # can it be checked out then swap its contents?

        nipyapi.versioning.save_flow_ver(
            process_group=process_group,
            registry_client=reg_client,
            bucket=bucket,
            flow_id=flow.identifier,
            comment=f"Updated {group} group",
        )
    else:
        nipyapi.versioning.save_flow_ver(
            process_group=process_group,
            registry_client=reg_client,
            bucket=bucket,
            flow_name=group,
            desc=f"{group} group",
            comment="Initial commit",
        )
except AttributeError as e:
    log.error(f"Cannot find unique process group named '{group}'")
except ValueError as e:
    log.error(e)

Can I somehow "check out" the flow, modify it, and save it back to the registry?

Had a look at fdlc.py but couldn't see anything obvious.

Thanks for any help!

Method to diff two flow versions in Registry 0.1.0

The upstream version of NiFi-Registry has introduced version differencing support, it would be useful to also have this function in the current version. Required for #31

error while create a PG under another PG

• Nipyapi version: 0.10.3
• NiFi version: 1.7.1
• Python version: 3.6.6
• Operating System: Windows 10 Pro

Description

I am not able to create a process group inside another process group. I can create it under a root though.

Example:
/root/level1 PG (created)
/root/level1/level2 (error)

What I Did

nipyapi.canvas.create_process_group('level2', 'level1', '100,100')

File "C:\Python\Python36\lib\site-packages\nipyapi\nifi\rest.py", line 268, in POST
body=body)
File "C:\Python\Python36\lib\site-packages\nipyapi\nifi\rest.py", line 224, in request
raise ApiException(http_resp=r)
nipyapi.nifi.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 01 Oct 2018 11:14:56 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Type': 'text/plain', 'Vary': 'Accept-Encoding', 'Content-Length': '68', 'Server': 'Jetty(9.4.3.v20170317)'})
HTTP response body: A revision of 0 must be specified when creating a new Process group.

Trouble Accessing NiFi via Authenticated Proxy

• Nipyapi version: 0.8.0
• NiFi version: 1.6.0
• NiFi-Registry version: N/A
• Python version: 3.6.3
• Operating System: CentOS 7.4.1708

Description

We are currently running NiFi behind an NGINX proxy with basic http authentication setup (NIFI itself is not secure). Accessing the interface manually works just fine. However we cant seem to get nipyapi to authenticate against the NGINX basic auth.

We are not sure if we are doing something wrong, misunderstood the configuration options, or perhaps this isn't currently achievable with nipyapi.

What I Did

We ran the following script against the NGINX presented url that proxies to the backed of http://nifi:8080 and set the configuration options of the NGINX auth appropriately.

#!/usr/bin/env python3
import nipyapi
nifi = 'https://localhost/nifi-api'
nipyapi.nifi.configuration.verify_ssl = False
nipyapi.config.nifi_config.host = nifi
nipyapi.nifi.configuration.username = 'username'
nipyapi.nifi.configuration.password = 'password'
print(nipyapi.system.get_nifi_version_info())

The script responds with error 401

nipyapi.nifi.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.13.12', 'Content-Length': '196', 'Connection': 'keep-alive', 'Content-Type': 'text/html', 'Date': 'Fri, 11 May 2018 14:48:02 GMT', 'WWW-Authenticate': 'Basic realm="Restricted Access"'})
HTTP response body: <html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>

Urgency

We are currently in development, nothing is in production today. We started to use nipyapi to automate some initial deployment tasks. We are held up a bit moving forward with some aspects of development but can more or less work around them for the time being.

Support Mnemonic pathing

As a user, it would be helpful to be able to address an object via a path structure using friendly names, rather than relying on UUIDs.

This would require a method for resolving object names in categories, handling duplicates (or banning them).
It would also be good to do this in such a way that IDE's can tab-complete the paths using object inspection.

Suggested by aperepel in #31

Retrieve and update properties of a controller attached to a process group

• Nipyapi version: 0.9.1
• NiFi version: 1.6.0
• NiFi-Registry version: 0.1
• Python version: 3.6
• Operating System: linux ( mint/ubuntu)

Description

Try to set the password on a controller that is at the process group level. I've done it for processors that need a password. I'm pushing this processor group up along with embedded controllers via nifi registry and so it strips out my password..and i'd like to just set it via the api either directly or set the ${MYPASSWORD} style.

What I Did

Grabbed info about the processor.

pg = nipyapi.canvas.get_process_group('myGroup')
ps = nipyapi.nifi.ProcessGroupsApi().get_processors(pg.id)

In both the process group and processor objects I'm not able to see a list of controllers. I've looked through the api but i don't see a way to get a list of controllers associated with process group.

I know the processor in question and i see put-db-record-dcbp-service : "e86a4746-fc57-32b3-a5e5-feffeb8b2ddc" ... but that e86 number is not the ID i see on the controller. Maybe there is a way to go from this number to the controller id and then I can update the controller. Didn't see anything in the demo about how to do that, but i'm sure i can figure that out.

Urgency

Not super urgent, might be able to go through the raw api. i've not dug super deep yet, so its possible i'm just missing something. Thanks so much for this project! Hopefully i'm not just missing something obvious.

UX: Promote a given version of the flow across registry instances

As a deployer, I want to promote a specific version of the flow from RegistryDEV to RegistryQA. This could be a convenience command wrapping the download/publish command chain.

Fix Configuration section example

• Nipyapi version:
• Python version:
• Operating System:

Description

Fix Configuration section example in the doc. Not working now properly

support default URL for NiFi to talk to Registry in config.py

As a user, it would be useful to centrally set the URL that NiFi should use when connecting to Registry, as it may differ from that which the host running nipyapi or some GUI user might use.

This is particularly important in secured environments where certificates and other items will likely have set hostnames