[GUIDE] Running PyFunceble in Conda Virtual Environments about pyfunceble

[mitchellkrogza]

You really should try Mini(Conda) it takes all of < 1 minute to create an environment with PyFunceble and start testing. I'll never use Python on any Linux distro anymore without using Conda environments as it just works and specifying and changing python versions takes seconds to do. I have mentioned to @funilrys about creating a docker image that people can use which is probably something coming in the near future. For now just try Mini(Conda) we still don't experience any freezing on any of our many projects that use PyFunceble so despite me trying to make it freeze I have not been successful myself in replicating that issue.

[mitchellkrogza]

Here is a simply bash script which will create the environment for you, run your test and delete the environment when done. @maravento @funilrys

#!/bin/bash

# ---------------------------------------------------------------------------------------------------
# mini(Conda) PyFunceble Testing Environment
# This will create a new environment every time and delete the environment when finished testing 
# Created By: Mitchell Krog - https://github.com/mitchellkrogza
# PyFunceble Created By: Nissar Chababy - https://guthub.com/funilrys
# ---------------------------------------------------------------------------------------------------

# ---------------------------------------------------------------------------------------------------
# FIRST INSTALL Mini(Conda) - Do this manually, this script will not do it for you
#                             and only needs to be done once
# 1. Add Conda Path to .bashrc (add line below to bottom of bashrc)
#
#    export PATH="${HOME}/miniconda/bin:${PATH}"
#
# 2. Reload your bashrc
#
#    source .bashrc
#
# 3. Download Conda
#
#    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
#
# 4. Install Conda
#
#    bash miniconda.sh -b -p ${HOME}/miniconda
#
# 5. Setup Conda
#
#    hash -r
#    conda config --set always_yes yes --set changeps1 n
# ---------------------------------------------------------------------------------------------------

# --------------------------------------------------
# YOUR SETTINGS - ADJUST THINGS ONLY IN THIS BLOCK
# --------------------------------------------------
TESTFOLDER=/home/username/PyFuncebleTests/DummyDemo/
TESTINGFILE=/home/username/MyTestLists/input.lst
ENVIRONMENTNAME="dummydemo"
PYFUNCEBLEPROCESSES=50
# Set Your Database Type mysql / mariadb
DATABASETYPE="mysql"
# --------------------------------------------------
# END SETTINGS
# --------------------------------------------------

# -----------
# Setup Conda
# -----------

echo "Setting up Conda"
export PATH="${HOME}/miniconda/bin:${PATH}"

# ------------
# Update Conda
# ------------

echo "Updating Conda"
conda update -q conda

# ------------------------
# Create Conda Environment
# ------------------------

conda create -q -n ${ENVIRONMENTNAME} python="3.7.4"

# -----------------------------
# To keep Environment up to date with latest Python, assuming you disable this script deleting the environment, then the following command will always upgrade Python
# -----------------------------

#conda update python

# --------------------
# Activate Environment
# --------------------

echo "Activating Test Environment"
source activate ${ENVIRONMENTNAME}

# --------------------------------------
# Install / Upgrade Requirements and Pip 
# --------------------------------------

echo "Upgrading Environment"

pip install --upgrade pip
pip install PyFunceble --upgrade
pip install domain2idna --upgrade
pip install ultimate-hosts-blacklist-whitelist --upgrade

# --------------------
# Show Version Numbers
# --------------------

echo "------------"
echo "VERSION INFO"
echo "------------"
python -VV
pip --version
domain2idna -v
ultimate-hosts-blacklist-whitelist -v
PyFunceble -v
echo "------------"

# ----------------
# Setup PyFunceble
# ----------------

cd ${TESTFOLDER}
export PYFUNCEBLE_CONFIG_DIR=${TESTFOLDER}

# --------------------
# Run PyFunceble Tests
# --------------------

# NOTE: Because of the MySQL / Mariabd database selection the first time you run this you need to manually answer the questions for the Database Connection
# PyFunceble will then create the .pyfunceble-env file containing the database connection information and won't ask again when you run this a second time
# Set your database up using PhpMyAdmin 

echo "Starting PyFunceble Tests"

PyFunceble --database-type ${DATABASETYPE} --idna -ex --multiprocess --processes ${PYFUNCEBLEPROCESSES} --plain -f ${TESTINGFILE}

# This Command Line below is a Simple One Line Test of github.com - useful for testing this script without running a full test
# Disable above line and enable this line below if you want to enable this quick test
#PyFunceble --idna -ex --multiprocess --processes ${PYFUNCEBLEPROCESSES} --plain -d github.com


# --------------------
# Deactive Environment
# --------------------

echo "Deactivating Test Environment"

conda deactivate

# ------------------
# Delete Environment
# ------------------
# Does not Delete any of your Data only the Environment
# This is NOT really necessary but can be used if you want to create a new environment every time

echo "Deleting Test Environment"

conda remove --name ${ENVIRONMENTNAME} --all

# -----------------------
# List Conda Environments
# -----------------------

echo "Listing Conda Environments"

conda env list

# ----------------------
# Exit With Error Number
# ----------------------

exit ${?}

[mitchellkrogza]

Sorry you can comment out the blacklist-whitelist installation I use that in my scripts

No sudo at all this is created as myscript.sh (whatever you want to name it) in your own user folder everything runs as your user

We centralized whitelisting which is now much more advanced if you go look at the project

[mitchellkrogza]

Until one day when a containerised version is produced this simple bash just about does that and perfectly too

[funilrys]

Nothing to add 👍 Can you make it a PR in the installation section? (As a new installation method of course)

[mitchellkrogza]

Have you actually tried running it in Mini(Conda) and isolating it as I
have suggested numerous times?

[mitchellkrogza]

Can rule out software-properties-common and python-pip for a start. But I promise you follow my guide I posted and in < 2 minutes you can have it running isolated in its own environment

[mitchellkrogza]

Inside Conda you use just pip Your version of python for the environment is specified when you create it thereafter just pip is used

[mitchellkrogza]

As for your earlier question about freezing and other packages that may be conflicting. A thought just popped into my head this morning. Have you checked AppArmor is not affecting things on your system? Some of my strangest problems on Ubuntu in the past have led to AppArmor configs

[mitchellkrogza]

Not at present.

When I run tests it does indeed consume one core for a while but then it jumps to another core and keeps doing this. I think there is probably ways in Python to balance it across the cores better. I am sure multiprocessing will be tweaked as time goes by. I am sure @funilrys can review the way multi uses cores when he next has free time.

I've however never had mine run into using swap at all.

For the majority of my tests I am keeping processes to 50-100 so as not to ever drag the machine down and allow me to happily continue doing other things.

[mitchellkrogza]

Does test IPs too

[mitchellkrogza]

What is wrong with the environment name? It's merely a placeholder which is meant to be changed. dummydemo is not aimed nor intended to be funny or insulting. By all means rewrite it however you want it was as I stated ... An example, a basic guide and is to be adapted to be used in your own environments 🤔

[mitchellkrogza]

I spend time writing an EXAMPLE for you can you can't even give a simple thank you for my time and effort merely pointing at what you perceive as bugs. I never said I was providing a 100% piece of software that is set and forget. You asked for an example script and I provided such.

[spirillen]

Hey @mitchellkrogza and @funilrys I think you have closed this a bit prematurely

As @funilrys suggested this in the docs in #39 (comment) but the docs still not showing this very very very cool approach
https://pyfunceble.readthedocs.io/en/dev/search.html?q=conda&check_keywords=yes&area=default#

which have slipped my attention until now, I think a it should have a pin, until it is covered in the docs.

I'm going to test it for sure 😃