elyra-ai / examples Goto Github PK
View Code? Open in Web Editor NEWA repository of examples for Elyra (https://github.com/elyra-ai/elyra)
License: Apache License 2.0
A repository of examples for Elyra (https://github.com/elyra-ai/elyra)
License: Apache License 2.0
Version 2.3 introduces examples that are not compatible with Elyra 2.2.x. We do want to preserve the legacy examples for users who are running one of the 2.2.x legacy versions.
Complements #9 and uses the same pipeline
As a user I want to be able to run a small pipeline to confirm that
In support of those tasks we need publish a ready-to-use pipeline that takes very little time to execute but essentially confirms that the user has a working setup.
Some of the example artifacts currently fail linting, which triggers warnings if language server support for Python is enabled.
In KFP v1, any value can be given for the type
field of an input. This can be problematic during pipeline compile
if parameters are passed to a pipeline. An InconsistentTypeException
will be thrown if the type hint of a parameter does not match the type of the input defined in the component YAML.
For example, in the inputs for the Download Data example component, the values of string
and URI
are given as types for the inputs. While it is clear that these types should be strings, an InconsistentTypeException
is thrown during compile
for a pipeline with a parameter type-hinted as str
in the Python DSL because KFP defines its string type as String
with a capital S.
We should update all our example component resources to only use KFP base types so that these components continue to work 'out of the box' when support for pipeline parameters is available in Elyra (elyra-ai/elyra#3001).
Considering that the purpose of the introductory Elyra tutorial pipelines is to enable new users to become familiar with pipeline features it shouldn't be necessary to use notebooks that take (in total) several minutes to run. The assets, which were originally introduced for https://github.com/elyra-ai/examples/tree/main/pipelines/dax_noaa_weather_data are currently also used by these pipelines:
The notebooks should be replaced with dummy versions that run in seconds and don't require package prerequisites that might break over time.
Some of the examples need to be updated to reflect the changes/additions in Elyra 2.0
to document
When following the Running pipelines in a local environment tutorial to step Running a notebook pipeline locally, a user may see the following error message:
The error details is:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 191, in process
papermill.execute_notebook(
File "/usr/local/lib/python3.9/site-packages/papermill/execute.py", line 118, in execute_notebook
raise_for_execution_errors(nb, output_path)
File "/usr/local/lib/python3.9/site-packages/papermill/execute.py", line 230, in raise_for_execution_errors
raise error
papermill.exceptions.PapermillExecutionError:
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-2-dd0d2a848dc3> in <module>
1 # Define required imports
----> 2 import pandas as pd
3 import numpy as np
4 import sys
5 import re
ModuleNotFoundError: No module named 'pandas'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 74, in process
operation_processor.process(operation)
File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 197, in process
raise RuntimeError(f'Internal error executing {filepath}: {ex}') from ex
RuntimeError: Internal error executing /Users/xuanyu/examples/pipelines/hello_world/Part 1 - Data Cleaning.ipynb:
---------------------------------------------------------------------------
Exception encountered at "In [2]":
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-2-dd0d2a848dc3> in <module>
1 # Define required imports
----> 2 import pandas as pd
3 import numpy as np
4 import sys
5 import re
ModuleNotFoundError: No module named 'pandas'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/handlers.py", line 89, in post
response = await PipelineProcessorManager.instance().process(pipeline)
File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor.py", line 70, in process
res = await asyncio.get_event_loop().run_in_executor(None, processor.process, pipeline)
File "/usr/local/Cellar/[email protected]/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/elyra/pipeline/processor_local.py", line 79, in process
raise RuntimeError(f'Error processing operation {operation.name}.') from ex
RuntimeError: Error processing operation Part 1 - Data Cleaning.
The reason is that the package pandas
was not installed. When we open the notebook Part 1 - Data Cleaning.ipynt
, we'll see this screen:
Installing pandas
by running pip3 install pandas
can resolve this problem.
It is a small thing and can be easily fixed. However, I don't find anywhere in the documentation of installing Elyra or the documentation of Running pipelines in a local environment that says that pandas
is one of the prerequites.
So, we may want to tell a user at Prerequisites section that they need to make sure that pandas
along with other packages like numpy
should be there before running the pipeline.
Environment:
Steps:
pip install --upgrade "elyra>=2.0.1" && jupyter lab build
.pipeline
files to enforce best practices.lint
taskvNext requires a migration.
The pipeline should illustrate the content in https://elyra.readthedocs.io/en/latest/recipes/visualizing-output-in-the-kfp-ui.html and be ready to run.
Hello everyone, im new using elyra, when I want to run the load_data.ipynb
(https://github.com/elyra-ai/examples/tree/main/pipelines/introduction-to-generic-pipelines ) example I get this error. I am running it on a TLJH (The Littlest JupyterHub) instance, I have had no problems with other extensions.
Traceback (most recent call last):
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/local/processor_local.py", line 228, in process
papermill.execute_notebook(filepath, filepath, **additional_kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/papermill/execute.py", line 113, in execute_notebook
nb = papermill_engines.execute_notebook_with_engine(
File "/opt/tljh/user/lib/python3.9/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/papermill/engines.py", line 367, in execute_notebook
cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/elyra_engine.py", line 83, in execute_managed_notebook
return PapermillNotebookClient(nb_man, **final_kwargs).execute(**kernel_kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/papermill/clientwrap.py", line 46, in execute
info_msg = self.wait_for_reply(self.kc.kernel_info())
File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/asynchronous/client.py", line 16, in _
msg_id = meth(self, *args, **kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/client.py", line 718, in kernel_info
self.shell_channel.send(msg)
File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/channels.py", line 263, in send
self.session.send(self.socket, msg)
File "/opt/tljh/user/lib/python3.9/site-packages/jupyter_client/session.py", line 853, in send
stream.send_multipart(to_send, copy=copy)
File "/opt/tljh/user/lib/python3.9/site-packages/zmq/_future.py", line 199, in send_multipart
return self._add_send_event('send_multipart', msg=msg, kwargs=kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/zmq/_future.py", line 354, in _add_send_event
f = future or self._Future()
File "/opt/tljh/user/lib/python3.9/asyncio/events.py", line 642, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'asyncio_1'.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/local/processor_local.py", line 100, in process
operation_processor.process(operation, elyra_run_name)
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/local/processor_local.py", line 237, in process
self.log_and_raise(file_name, ex)
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/local/processor_local.py", line 180, in log_and_raise
raise RuntimeError(f"({file_name}): {truncated_msg}") from ex
RuntimeError: (load_data.ipynb): There is no current event loop in thread 'asyncio_1'.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/tljh/user/lib/python3.9/site-packages/tornado/web.py", line 1713, in _execute
result = await result
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/handlers.py", line 160, in post
response = await PipelineProcessorManager.instance().process(pipeline)
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/processor.py", line 166, in process
res = await asyncio.get_event_loop().run_in_executor(None, processor.process, pipeline)
File "/opt/tljh/user/lib/python3.9/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/opt/tljh/user/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
future.result()
File "/opt/tljh/user/lib/python3.9/asyncio/futures.py", line 201, in result
raise self._exception
File "/opt/tljh/user/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/tljh/user/lib/python3.9/site-packages/elyra/pipeline/local/processor_local.py", line 108, in process
raise RuntimeError(f"Error processing operation {operation.name} {str(ex)}") from ex
RuntimeError: Error processing operation load_data (load_data.ipynb): There is no current event loop in thread 'asyncio_1'.
What can I do to solve it?
this my environment
------- OS ----------------------
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
------- Python Version ---------
3.9.7
------- Node JS Version --------
v14.10.0
------- Elyra Version ------------
elyra==3.14.3
elyra-code-snippet-extension==3.14.3
elyra-code-viewer-extension==3.14.3
elyra-examples-kfp-catalog==0.1.0
elyra-pipeline-editor-extension==3.14.3
elyra-python-editor-extension==3.14.3
elyra-server==3.14.3
------- Jupyter --------
jupyter-app-launcher==0.1.5
jupyter_client==7.4.9
jupyter-contrib-core==0.4.0
jupyter_core==5.1.5
jupyter-events==0.6.3
jupyter-lsp==1.5.1
jupyter-nbextensions-configurator==0.6.1
jupyter_packaging==0.12.3
jupyter-resource-usage==0.6.4
jupyter-server==1.23.5
jupyter-server-mathjax==0.2.6
jupyter-server-proxy==3.2.2
jupyter-telemetry==0.1.0
jupyterhub==1.5.1
jupyterlab==3.5.3
jupyterlab-git==0.41.0
jupyterlab-lsp==3.10.2
jupyterlab-pygments==0.2.2
jupyterlab_server==2.19.0
jupyterlab-widgets==1.1.1
nteract-on-jupyter==2.1.3
-------- Server Extensions -------
Config dir: /opt/tljh/user/etc/jupyter
elyra enabled
- Validating elyra...
elyra OK
jupyter_lsp enabled
- Validating jupyter_lsp...
jupyter_lsp 1.5.1 OK
jupyter_server_proxy enabled
- Validating jupyter_server_proxy...
jupyter_server_proxy OK
jupyter_app_launcher enabled
- Validating jupyter_app_launcher...
jupyter_app_launcher 0.1.5 OK
jupyter_resource_usage enabled
- Validating jupyter_resource_usage...
jupyter_resource_usage 0.6.4 OK
jupyter_server_mathjax enabled
- Validating jupyter_server_mathjax...
jupyter_server_mathjax OK
jupyterlab enabled
- Validating jupyterlab...
jupyterlab 3.5.3 OK
jupyterlab_git enabled
- Validating jupyterlab_git...
jupyterlab_git 0.41.0 OK
nbclassic enabled
- Validating nbclassic...
nbclassic 0.4.8 OK
nbdime enabled
- Validating nbdime...
nbdime 3.1.1 OK
nbgitpuller enabled
- Validating nbgitpuller...
nbgitpuller 1.1.1 OK
notebook_shim enabled
- Validating notebook_shim...
notebook_shim OK
Config dir: /usr/local/etc/jupyter
----------- Lab Extensions -----------
JupyterLab v3.5.3
/opt/tljh/user/share/jupyter/labextensions
jupyter_app_launcher v0.1.5 enabled OK (python, jupyter_app_launcher)
jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
nbdime-jupyterlab v2.1.1 enabled OK
@jupyter-widgets/jupyterlab-manager v3.1.1 enabled OK (python, jupyterlab_widgets)
@jupyterlab/git v0.41.0 enabled OK (python, jupyterlab-git)
@jupyterlab/server-proxy v3.2.2 enabled OK
@krassowski/jupyterlab-lsp v3.10.2 enabled OK (python, jupyterlab-lsp)
@elyra/metadata-extension v3.14.3 enabled OK
@elyra/theme-extension v3.14.3 enabled OK
@elyra/scala-editor-extension v3.14.3 enabled OK
@elyra/python-editor-extension v3.14.3 enabled OK
@elyra/r-editor-extension v3.14.3 enabled OK
@elyra/script-debugger-extension v3.14.3 enabled OK
@elyra/pipeline-editor-extension v3.14.3 enabled OK
@elyra/code-viewer-extension v3.14.3 enabled OK
@elyra/code-snippet-extension v3.14.3 enabled OK
@jupyter-server/resource-usage v0.6.4 enabled OK (python, jupyter-resource-usage)
Other labextensions (built into JupyterLab)
app dir: /opt/tljh/user/share/jupyter/lab
Thanks for your time
An upcoming release of Kubeflow adds support for [component] registry hosts, which implement an API that clients can use to work with components.
Requires machine-learning-exchange/mlx#331
To help novice users we should provide a basic Hello World
pipeline example along with instructions. Upon completion of this (5 minutes or less) tutorial, a user will have learned how to
Other tutorials, such as Hello Kubeflows Pipelines World
, can build on this tutorial, and would cover Kubeflow Pipelines specific tasks, such as configuring a runtime, monitoring a run, etc.
The tutorial should be similar to https://github.com/elyra-ai/examples/tree/master/pipelines/hello_world_kubeflow_pipelines.
ODH documentation: https://opendatahub.io/docs.html
ODH documentation source: https://gitlab.com/opendatahub/opendatahub.io
Similar to the ones we already have for local execution and execution on Kubeflow Pipelines.
https://github.com/elyra-ai/examples/blob/main/pipelines/dax_noaa_weather_data/Part%202%20-%20Data%20Analysis.ipynb
fails to execute, likely due to changes in an upstream dependency:
D 16:25:40.421] template_paths: /usr/local/share/jupyter/nbconvert/templates/lab:/usr/local/share/jupyter/nbconvert/templates/base:/root/.local/share/jupyter:/root/.local/share/jupyter/nbconvert/templates:/root/.local/share/jupyter/nbconvert/templates/compatibility:/usr/local/share/jupyter:/usr/local/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates/compatibility:/usr/share/jupyter:/usr/share/jupyter/nbconvert/templates:/usr/share/jupyter/nbconvert/templates/compatibility:/usr/local/share/jupyter/nbconvert/templates
[I 16:25:40.722] 'analyze_NOAA_weather_data':'Part 2 - Data Analysis' - Part 2 - Data Analysis-output.ipynb converted to Part 2 - Data Analysis.html (0.857 secs)
[D 16:25:40.767]
http://cloning1.fyre.ibm.com:30887/
"PUT /ptitzler-test/analyze_NOAA_weather_data-1003092202/Part%202%20-%20Data%20Analysis.ipynb HTTP/1.1" 200 0
[I 16:25:40.768] 'analyze_NOAA_weather_data':'Part 2 - Data Analysis' - uploaded Part 2 - Data Analysis-output.ipynb to bucket: ptitzler-test object: analyze_NOAA_weather_data-1003092202/Part 2 - Data Analysis.ipynb (0.045 secs)
[D 16:25:40.818]
http://cloning1.fyre.ibm.com:30887/
"PUT /ptitzler-test/analyze_NOAA_weather_data-1003092202/Part%202%20-%20Data%20Analysis.html HTTP/1.1" 200 0
[I 16:25:40.818] 'analyze_NOAA_weather_data':'Part 2 - Data Analysis' - uploaded Part 2 - Data Analysis.html to bucket: ptitzler-test object: analyze_NOAA_weather_data-1003092202/Part 2 - Data Analysis.html (0.051 secs)
Traceback (most recent call last):
File "bootstrapper.py", line 697, in <module>
main()
File "bootstrapper.py", line 687, in main
file_op.execute()
File "bootstrapper.py", line 367, in execute
raise ex
File "bootstrapper.py", line 352, in execute
papermill.execute_notebook(notebook, notebook_output, kernel_name=kernel_name)
File "/usr/local/lib/python3.8/site-packages/papermill/execute.py", line 122, in execute_notebook
raise_for_execution_errors(nb, output_path)
File "/usr/local/lib/python3.8/site-packages/papermill/execute.py", line 234, in raise_for_execution_errors
raise error
papermill.exceptions.PapermillExecutionError:
---------------------------------------------------------------------------
Exception encountered at "In [8]":
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Input In [8], in <cell line: 12>()
10 # Zoom in on a year and plot temperature data converted to a daily frequency
11 plt.subplot(1, 2, 2)
---> 12 data['2017'][TEMP_COL].asfreq('D').plot()
13 plt.title('Daily Temperature in 2017')
14 plt.ylabel('Temperature (F)')
File /usr/local/lib/python3.8/site-packages/pandas/plotting/_core.py:949, in PlotAccessor.__call__(self, *args, **kwargs)
946 label_name = label_kw or data.columns
947 data.columns = label_name
--> 949 return plot_backend.plot(data, kind=kind, **kwargs)
File /usr/local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py:61, in plot(data, kind, **kwargs)
59 kwargs["ax"] = getattr(ax, "left_ax", ax)
60 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 61 plot_obj.generate()
62 plot_obj.draw()
63 return plot_obj.result
File /usr/local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py:274, in MPLPlot.generate(self)
272 self._add_table()
273 self._make_legend()
--> 274 self._adorn_subplots()
276 for ax in self.axes:
277 self._post_plot_logic_common(ax, self.data)
File /usr/local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py:468, in MPLPlot._adorn_subplots(self)
466 all_axes = self._get_subplots()
467 nrows, ncols = self._get_axes_layout()
--> 468 _handle_shared_axes(
469 axarr=all_axes,
470 nplots=len(all_axes),
471 naxes=nrows * ncols,
472 nrows=nrows,
473 ncols=ncols,
474 sharex=self.sharex,
475 sharey=self.sharey,
476 )
478 for ax in self.axes:
479 if self.yticks is not None:
File /usr/local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/tools.py:331, in _handle_shared_axes(axarr, nplots, naxes, nrows, ncols, sharex, sharey)
326 if ncols > 1:
327 for ax in axarr:
328 # only the first column should get y labels -> set all other to
329 # off as we only have labels in the first column and we always
330 # have a subplot there, we can skip the layout test
--> 331 if ax.is_first_col():
332 continue
333 if sharey or len(ax.get_shared_y_axes().get_siblings(ax)) > 1:
AttributeError: 'AxesSubplot' object has no attribute 'is_first_col'
This impacts many of the introductory Elyra pipelines: #111
Details TBD
Can you please help me to read the component label and the node id from yaml, and pass that as an argument to .py file?
... to use relative paths pending delivery of elyra-ai/elyra#861
To reflect the new support for Python nodes. Since there is no fundamental difference between a notebook node and a Python node it should be sufficient to
Requires Elyra 1.2
Follow-up to #9. If a user launches binder the pipelines tutorial artifacts should be included by default, eliminating the need to download them from https://github.com/elyra-ai/examples/.
Can Elyra users please gather and link in more examples of how to use for LOCAL-ONLY file-based pipelines using Elyra Pipeline manager?
The examples in this repo really emphasize Kubernetes and Airflow cloud-based applications.
However, Elyra is more general than cloud-only based visual pipeline management.
More LOCAL-ONLY examples will be very helpful.
Thank you.
The tutorials cover only generic pipelines
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.