GithubHelp home page GithubHelp logo

tardunge / prefect-spark-on-k8s-operator Goto Github PK

View Code? Open in Web Editor NEW
4.0 4.0 1.0 939 KB

Prefect integrations for orchestrating and monitoring apache spark jobs on kubernetes using spark-on-k8s-operator.

License: Apache License 2.0

Python 100.00%
kubernetes prefect prefect-kubernetes spark-on-k8s-operator

prefect-spark-on-k8s-operator's People

Contributors

abhishekk3 avatar dependabot[bot] avatar tardunge avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

abhishekk3

prefect-spark-on-k8s-operator's Issues

”TypeError: Object of type 'FieldInfo' is not JSON serializable“ with prefect-2.14.13 prefect-kubernetes-0.2.3 in python3.8

Traceback (most recent call last):
File "test_spark_application.py", line 8, in
from prefect_spark_on_k8s_operator import SparkApplication
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect_spark_on_k8s_operator/init.py", line 2, in
from prefect_spark_on_k8s_operator.app import ( # noqa F401
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect_spark_on_k8s_operator/app.py", line 39, in
class SparkApplication(JobBlock):
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/main.py", line 282, in new
cls = super().new(mcs, name, bases, new_namespace, **kwargs)
File "/usr/lib/python3.8/abc.py", line 85, in new
cls = super().new(mcls, name, bases, namespace, **kwargs)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/utilities/dispatch.py", line 99, in _register_subclass_of_base_type
register_type(cls)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/utilities/dispatch.py", line 153, in register_type
key = get_dispatch_key(cls)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/utilities/dispatch.py", line 76, in get_dispatch_key
dispatch_key = dispatch_key()
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/blocks/core.py", line 315, in dispatch_key
return block_schema_to_key(cls._to_block_schema())
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/prefect/blocks/core.py", line 487, in to_block_schema
fields = cls.schema()
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/main.py", line 664, in schema
s = model_schema(cls, by_alias=by_alias, ref_template=ref_template)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 188, in model_schema
m_schema, m_definitions, nested_models = model_process_schema(
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 582, in model_process_schema
m_schema, m_definitions, nested_models = model_type_schema(
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 623, in model_type_schema
f_schema, f_definitions, f_nested_models = field_schema(
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 249, in field_schema
s, schema_overrides = get_field_info_schema(field)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 217, in get_field_info_schema
schema
['default'] = encode_default(field.default)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/schema.py", line 996, in encode_default
return pydantic_encoder(dft)
File "/home/xxxx/pyvenv/lib/python3.8/site-packages/pydantic/v1/json.py", line 90, in pydantic_encoder
raise TypeError(f"Object of type '{obj.class.name}' is not JSON serializable")
TypeError: Object of type 'FieldInfo' is not JSON serializable

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.