bogdancojocar / medium-articles

Repo for all my code on the articles I post on medium

Repository for Medium articles

• jupyter_kafka - How to do a sentiment analysis in realtime using the Jupyter notebook, Kafka and NLTK
• kafka_nlp - Building a realtime NLP pipeline using Kafka and spaCy
• livy_batch_emr - How to do better deployments of spark jobs to aws emr using apache livy
• pandas_validation - How to do column validation with pandas
• porto_seguro_spark - Safe driver prediction using PySpark and Logistic Regression
• pyspark-project-template - How to setup the Python and Spark environment for development, with good software engineering practices
• titanic_spark - Realtime prediction using Spark Structured Streaming, XGBoost and Scala
• titanic_xgboost - PySpark ML and XGBoost full integration tested on the Kaggle Titanic dataset
• scala_notebook_test - How to run Scala and Spark in the Jupyter notebook
• mlflow-automl - How to build an integration between AutoML and MLFlow
• realtime_kafka - Building a real-time prediction pipeline using Spark Structured Streaming and Microservices
• realtime_fraud_detection - How to build a real-time fraud detection pipeline using Faust and MLFlow
• terraform_eks_spark - How to run a PySpark job in Kubernetes (AWS EKS)
• s3a_spark - How to read parquet data from S3 using the S3A protocol and temporary credentials in PySpark
• pyflink_riverml - Building a Credit Card Fraud Detection Online Training Pipeline with River ML and Apache Flink
• cdktf_azure_sentiment_ml - Building a Serverless Azure ML Service Using Cognitive and CDKTF
• aks_seldon - Building a Health Entity labelling service using Azure Kubernetes Service, Seldon Core and Azure Cognitive
• neo4j_companies_knn - Predicting similar political donors for UK parties using graph data
• xgboost_docker - PySpark ML and XGBoost setup using a docker image
• xgboost_pyspark - PySpark integration with the native python package of XGBoost

You can stay up to date with the latest stories I post on medium here

medium-articles's Issues

Issues with XGBoostEstimator cell

Hi there,

first, thank you for a great Medium post and for this repository. I am new to PySpark, and am trying to run your code. It all works fine util I get to xgboost = XGBoostEstimator(... cell. When I try to run it, i get the following error message:

TypeError: 'JavaPackage' object is not callable

If you have any ides why this might be the case, or what I could do to resolve this issue, it would be very helpful.

Thanks,

Bojan

qubole/kinesis-sql is broken

qubole/kinesis-sql is actually discontinued. I've taken over the maintenance of it. I just released a new version for Spark 3.2. See here for details: https://github.com/roncemer/spark-sql-kinesis

The old project directs existing users to the repo which I'm maintaining (although they did misspell my name, LOL): https://github.com/qubole/kinesis-sql

TypeError: 'JavaPackage' object is not callable on trying to run XGBoostEstimator, which imports just OK

As I run the code

xgboost = XGBoostEstimator( featuresCol="features", labelCol="Survival", predictionCol="prediction" )

I got an error

TypeError Traceback (most recent call last)
in ()
1 xgboost = XGBoostEstimator(
2 # featuresCol="features",
----> 3 labelCol="Survival"
4 )

2 frames
/content/spark-2.4.8-bin-hadoop2.7/python/pyspark/ml/wrapper.py in _new_java_obj(java_class, *args)
65 java_obj = getattr(java_obj, name)
66 java_args = [_py2java(sc, arg) for arg in args]
---> 67 return java_obj(*java_args)
68
69 @staticmethod

TypeError: 'JavaPackage' object is not callable

[Question] Where is the data.csv for the realtime_fraud_detection example thanks

Much appreciated