[BUG] OOM in nightly SliRec and DKN about recommenders

[miguelgfierro]

Still the same error in nightly GPU: https://github.com/recommenders-team/recommenders/actions/runs/8372395934/job/22923293913

2024-03-21 09:53:25.709498: W external/local_tsl/tsl/framework/bfc_allocator.cc:497] ********____***_____*_____*****************___*____*____***************************************xxxxx
2024-03-21 09:53:25.709518: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at conv_grad_input_ops.cc:345 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[10000,1,10,300] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
_ test_slirec_quickstart_functional[recommenders/models/deeprec/config/sli_rec.yaml-tests/resources/deeprec/slirec-10-400-expected_values0-42] _

notebooks = ***'als_deep_dive': '/mnt/azureml/cr/j/b321b89c582248be83f964f9e412bd91/exe/wd/examples/02_model_collaborative_filtering...rk_movielens': '/mnt/azureml/cr/j/b321b89c582248be83f964f9e412bd91/exe/wd/examples/06_benchmarks/movielens.ipynb', ...***
output_notebook = 'output.ipynb', kernel_name = 'python3'
yaml_file = 'recommenders/models/deeprec/config/sli_rec.yaml'
data_path = 'tests/resources/deeprec/slirec', epochs = 10, batch_size = 400
expected_values = ***'auc': 0.7183***, seed = 42

    @pytest.mark.gpu
    @pytest.mark.notebooks
    @pytest.mark.parametrize(
        "yaml_file, data_path, epochs, batch_size, expected_values, seed",
        [
            (
                "recommenders/models/deeprec/config/sli_rec.yaml",
                os.path.join("tests", "resources", "deeprec", "slirec"),
                10,
                400,
                ***
                    "auc": 0.7183
                ***,  # Don't do logloss check as SLi-Rec uses ranking loss, not a point-wise loss
                42,
            )
        ],
    )
    def test_slirec_quickstart_functional(
        notebooks,
        output_notebook,
        kernel_name,
        yaml_file,
        data_path,
        epochs,
        batch_size,
        expected_values,
        seed,
    ):
        notebook_path = notebooks["slirec_quickstart"]
    
        params = ***
            "yaml_file": yaml_file,
            "data_path": data_path,
            "EPOCHS": epochs,
            "BATCH_SIZE": batch_size,
            "RANDOM_SEED": seed,
        ***
>       execute_notebook(
            notebook_path, output_notebook, kernel_name=kernel_name, parameters=params
        )

[SimonYansenZhao]

Could you provide more info about the GPU size used for nightly build? @miguelgfierro

[miguelgfierro]

T4: https://learn.microsoft.com/en-us/azure/virtual-machines/nct4-v3-series