Example PySpark project with standard configuration for:
- code style and analysis [flake8, pylint, PyCharm]
- static vulnerability analysis [bandit]
- unit tests [py.test]
- test coverage [coverage]
- documentation [sphinx]
$ pip3 install tox
- Install PyCharm
- Start a new project
- Start a new virtualenv in PyCharm based on Python3 under
File;Settings...;Project;Project Interpreter
and addtox
andpyspark
.
You can either copy the files in this template and rename as needed, or use it as an example to create your own files.
- In your project add your own 'pyspark_script.py'.
- Add pytests in a
tests/
directory - Add a
setup.py
- Run
Tools;Sphinx Quickstart
- Add a
tox.ini
- Run tox.ini