tf and CUDA recognise GPU, yet ray local instances start without available gpus about flower

[jafermarq]

Hi @psklavos1, If you try to run the code in examples/simulation-tensorflow with --num_gpus>0 do you experience the same problem?

[psklavos1]

Yes. The same problem persists. I don't want to trouble you with debugging. I would just like any points to examine in order to resolve the issue. Thank in advance for your help

…

On Thu, Oct 12, 2023, 11:26 PM Javier ***@***.***> wrote: Hi @psklavos1 <https://github.com/psklavos1>, If you try to run the code in examples/simulation-tensorflow <https://github.com/adap/flower/tree/main/examples/simulation-tensorflow> do you experience the same problem? — Reply to this email directly, view it on GitHub <#2502 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARVKWF6IU5F3F3I5VP5Q4DDX7BG6LAVCNFSM6AAAAAA54OVEYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRQGMYTSNRRGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jafermarq]

hmmm... did you add those print("Available GPUs:", tf.config.list_physical_devices("GPU")) in the same python file you later run start_simulation? Are you on Windows? (if yes, could you try WSL?)

[psklavos1]

Yes, they are at the same file(it is in the main.py). Essentialy, the code is an effort to emulate your videos on flower simulation, using tensor flow. I am already on Wsl.

…

On Thu, Oct 12, 2023, 11:59 PM Javier ***@***.***> wrote: hmmm... did you add those print("Available GPUs:", tf.config.list_physical_devices("GPU")) in the same python file you later run start_simulation? Are you on Windows? (if yes, could you try WSL?) — Reply to this email directly, view it on GitHub <#2502 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARVKWFZGLLGYVHZ4OLI32OLX7BK3DAVCNFSM6AAAAAA54OVEYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRQGM2TOOBYGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[psklavos1]

Is there any Idea of what to do to fix the problem. My Gpu is fully recognised when executing:
print("Available GPUs:", tf.config.list_physical_devices("GPU"))

and when trying to list:
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

But Ray still does not seem to initialize gpu device. Can I manually initialize to see the gpu in some way?

[jafermarq]

@alessiomora, do you know what could be causing this? @psklavos1 is trying to do simulation with TF in Window's WSL but his GPU isn't detected properly

[alessiomora]

I imagine that with 'num_gpus': 1.0 the problem persists, right? I see that the GPU memory logged in the issue is a bit more than 3GB (am I wrong?) and maybe 0.33 of that memory is not enough? However, the exemplary code should be very light in terms of memory footprint. Anyway, just to exclude that it is not a problem of memory.

[psklavos1]

Yes. I can verify the issue is the same with: 'num_gpus': 1.0.

If I run just a simple snippet like:

`import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy as np

np.random.seed(0)
X = np.random.rand(100, 2)
y = (X[:, 0] + X[:, 1] > 1).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = Sequential()
model.add(Dense(16, input_dim=2, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

with tf.device('/GPU:0'):
history = model.fit(X_train, y_train, epochs=100, batch_size=32, validation_data=(X_test, y_test))

loss, accuracy = model.evaluate(X_test, y_test)
print("Test loss:", loss)
print("Test accuracy:", accuracy)`

The Gpu is used with no errors whatsoever so the installation is correct. Although when running with Ray, the Gpu is not intantiated in the actorpool.

[Dknight27]

I had the same problem, but I was using pytorch.
output:
torch.cuda.is_available(): True
INFO flwr 2023-12-24 03:37:57,013 | app.py:178 | Starting Flower simulation, config: ServerConfig(num_rounds=10, round_timeout=None)
2023-12-24 03:37:59,156 INFO worker.py:1553 -- Started a local Ray instance.
INFO flwr 2023-12-24 03:38:01,192 | app.py:213 | Flower VCE: Ray initialized with resources: {'object_store_memory': 35014316851.0, 'memory': 71700072653.0, 'CPU': 40.0, 'node:10.139.50.89': 1.0}
INFO flwr 2023-12-24 03:38:01,192 | app.py:219 | Optimize your simulation with Flower VCE: https://flower.dev/docs/framework/how-to-run-simulations.html
INFO flwr 2023-12-24 03:38:01,192 | app.py:242 | Flower VCE: Resources for each Virtual Client: {'num_cpus': 1, 'num_gpus': 1.0}
WARNING flwr 2023-12-24 03:38:01,193 | ray_actor.py:144 | The ActorPool is empty. The system (CPUs=40.0, GPUs=0) does not meet the criteria to host at least one client with resources: {'num_cpus': 1, 'num_gpus': 1.0}. Lowering the client_resources could help.