Comments (17)
Good! I will create the PR and (re)activate some of the compatibilities. If that still works, then we can merge it :)
from waterlily.jl.
That's unfortunate. This week I will get access to LUMI, which I presume has a more updated ROCm version. I will look into it and let you know how that goes.
from waterlily.jl.
Yes, thank you :)
from waterlily.jl.
Hi Simon, thanks for reporting this. What ROCm version do you have installed in your system? I was recently able to run WaterLily on an AMD GPU, but because the ROCm version in that system was quite low (5.1.1) I had to force AMDGPU.jl to use its 0.4.15 version. Maybe something similar to this is happening?
Also, we could force WaterLily v1.1 to compile only upstream of a certain AMDGPU.jl version.
And it would also be helpful to see the full log of what packages changed versions when using AMDGPU
(if there were more than just WaterLily) - maybe there is an incompatibility with another package.
from waterlily.jl.
That wasn't the reason, I just updated all my drivers including rocm (5.7 + 6.1). I also don't think Pkg can even install different AMDGPU version based on available drivers, if I'm not mistaken.
You can actually reproduce it with this minimal setup (Julia 1.10.4):
]activate --temp
]add AMDGPU@0.9.6 WaterLily@1.1.0
from waterlily.jl.
What happens when you try to not specify the version, ie
]activate --temp
]add AMDGPU WaterLily
This works fine on my system, even though it installs AMDGPU v0.6.1. So maybe we should update our version compatibilities.
from waterlily.jl.
Can you please try my temporary fix in the fix_compat
branch and report back?
]
activate --temp
add WaterLily#fix_compat
add [email protected]
from waterlily.jl.
Yes, this seems to work :)
from waterlily.jl.
It does not work though :(
I'm getting lots of:
Reason: unsupported call through a literal pointer (call to jl_gc_run_pending_finalizers)
Reason: unsupported call to an unknown function (call to ijl_pop_handler)
I guess some kernel isn't setup correctly, although I would expect CUDA.jl to also caugh on something like jl_gc_run_pending_finalizers
....
Do you have an example that you have tested with AMDGPU that runs fine?
from waterlily.jl.
You can try the following MWE
using WaterLily
using AMDGPU
function tgv(p, backend; Re=1600, T=Float32)
L = 2^p; U = 1; Îș=Ï/L; Îœ = 1/(Îș*Re)
function uλ(i,xyz)
x,y,z = @. xyz/L*Ï # scaled coordinates
i==1 && return -U*sin(x)*cos(y)*cos(z) # u_x
i==2 && return U*cos(x)*sin(y)*cos(z) # u_y
return 0. # u_z
end
Simulation((L, L, L), (0, 0, 0), 1/Îș; U=U, uλ=uλ, Îœ=Îœ, T=T, mem=backend)
end
function main()
sim = tgv(5, ROCArray)
sim_step!(sim)
end
main()
I can correctly run this using ROCm/5.1.1, WaterLily.jl/1.1, AMDGPU.jl/0.4.15 on a Radeon Instinct MI50 32GB.
from waterlily.jl.
That actually kills the julia session đ
[email protected], WaterLily v1.1.0 and ROCm/6.1 on a 7900xtx.
The AMDGPU test seem to be getting much further on these new versions than any before!
from waterlily.jl.
The PR doesn't actually fix this issue.
from waterlily.jl.
Hey Simon, I have successfully run WaterLily on LUMI today (AMD MI250x), where ROCm/5.2.3. There are a couples of fixes for flows with bodies, but this example worked out of the box. This is my project environment right now
[21141c5a] AMDGPU v0.9.6
[0c68f7d7] GPUArrays v10.3.0
[63c18a36] KernelAbstractions v0.9.22
[90137ffa] StaticArrays v1.9.7
[ed894a53] WaterLily v1.2.0
And I get the following AMDGPU.jl info
julia> AMDGPU.versioninfo()
[ Info: AMDGPU versioninfo
âââââââââââââŹâââââââââââââââââââŹââââââââââââŹâââââââââââââââââââââââââââââââââââââââ
â Available â Name â Version â Path â
âââââââââââââŒâââââââââââââââââââŒââââââââââââŒâââââââââââââââââââââââââââââââââââââââ€
â + â LLD â - â /opt/rocm/llvm/bin/ld.lld â
â + â Device Libraries â - â /opt/rocm/amdgcn/bitcode â
â + â HIP â 5.2.21153 â /opt/rocm-5.2.3/lib/libamdhip64.so â
â + â rocBLAS â 2.44.0 â /opt/rocm-5.2.3/lib/librocblas.so â
â + â rocSOLVER â 3.18.0 â /opt/rocm-5.2.3/lib/librocsolver.so â
â + â rocALUTION â - â /opt/rocm-5.2.3/lib/librocalution.so â
â + â rocSPARSE â - â /opt/rocm-5.2.3/lib/librocsparse.so â
â + â rocRAND â 2.10.5 â /opt/rocm-5.2.3/lib/librocrand.so â
â + â rocFFT â 1.0.27 â /opt/rocm-5.2.3/lib/librocfft.so â
â + â MIOpen â 2.17.0 â /opt/rocm-5.2.3/lib/libMIOpen.so â
âââââââââââââŽâââââââââââââââââââŽââââââââââââŽâââââââââââââââââââââââââââââââââââââââ
[ Info: AMDGPU devices
ââââââŹâââââââŹâââââââââââââââââââââââââŹââââââââââââŹâââââââââââââ
â Id â Name â GCN arch â Wavefront â Memory â
ââââââŒâââââââŒâââââââââââââââââââââââââŒââââââââââââŒâââââââââââââ€
â 1 â â gfx90a:sramecc+:xnack- â 64 â 63.984 GiB â
ââââââŽâââââââŽâââââââââââââââââââââââââŽââââââââââââŽâââââââââââââ
I cannot try anything with ROCm/6.x though... which could be the problem.
from waterlily.jl.
I just found out that AMDGPU can be used on WSL2 :-O
Now I get:
julia> using AMDGPU
julia> function tgv(p, backend; Re=1600, T=Float32)
L = 2^p; U = 1; Îș=Ï/L; Îœ = 1/(Îș*Re)
function uλ(i,xyz)
x,y,z = @. xyz/L*Ï # scaled coordinates
i==1 && return -U*sin(x)*cos(y)*cos(z) # u_x
i==2 && return U*cos(x)*sin(y)*cos(z) # u_y
return 0. # u_z
end
Simulation((L, L, L), (0, 0, 0), 1/Îș; U=U, uλ=uλ, Îœ=Îœ, T=T, mem=backend)
end
tgv (generic function with 1 method)
julia> function main()
sim = tgv(5, ROCArray)
sim_step!(sim)
end
main (generic function with 1 method)
julia> main()
ERROR: Scalar indexing is disallowed.
Invocation of setindex! resulted in scalar indexing of a GPU array.
This is typically caused by calling an iterating implementation of a method.
Such implementations *do not* execute on the GPU, but very slowly on the CPU,
and therefore should be avoided.
from waterlily.jl.
Ok, that's progress! I think the only problem is now that you might be running Julia with single thread instead of auto
. Can you try julia -t auto ...
, or export JULIA_NUM_THREADS=auto
. Then validate that Threads.nthreads() > 1
. This behaviour will eventually be fixed by #133.
from waterlily.jl.
Yay, that makes the example work on WSL2 ubuntu!
I do notice now, that there seems to be a mismatch in the HIP version on windows:
Windows
julia> AMDGPU.versioninfo()
[ Info: AMDGPU versioninfo
âââââââââââââŹâââââââââââââââââââŹââââââââââââŹâââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
â Available â Name â Version â Path âŻ
âââââââââââââŒâââââââââââââââââââŒââââââââââââŒâââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
â + â LLD â - â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\ld.lld.exe âŻ
â + â Device Libraries â - â C:\\Users\\sdani\\.julia\\artifacts\\5ad5ecb46e3c334821f54c1feecc6c152b7b6a45\\amdgcn/ âŻ
â + â HIP â 5.7.32000 â C:\\WINDOWS\\SYSTEM32\\amdhip64.DLL âŻ
â + â rocBLAS â 4.1.2 â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\rocblas.dll âŻ
â + â rocSOLVER â 3.25.0 â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\rocsolver.dll âŻ
â + â rocALUTION â - â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\rocalution.dll âŻ
â + â rocSPARSE â - â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\rocsparse.dll âŻ
â + â rocRAND â 2.10.5 â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\rocrand.dll âŻ
â + â rocFFT â 1.0.27 â C:\\Program Files\\AMD\\ROCm\\6.1\\bin\\rocfft.dll âŻ
â - â MIOpen â - â - âŻ
âââââââââââââŽâââââââââââââââââââŽââââââââââââŽâââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
1 column omitted
[ Info: AMDGPU devices
ââââââŹââââââââââââââââââââââââââŹâââââââââââŹââââââââââââŹâââââââââââââ
â Id â Name â GCN arch â Wavefront â Memory â
ââââââŒââââââââââââââââââââââââââŒâââââââââââŒââââââââââââŒâââââââââââââ€
â 1 â AMD Radeon RX 7900 XTX â gfx1100 â 32 â 23.984 GiB â
â 2 â AMD Radeon(TM) Graphics â gfx1036 â 32 â 12.019 GiB â
ââââââŽââââââââââââââââââââââââââŽâââââââââââŽââââââââââââŽâââââââââââââ
WSL ubuntu
[ Info: AMDGPU versioninfo
âââââââââââââŹâââââââââââââââââââŹââââââââââââŹââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
â Available â Name â Version â Path â
âââââââââââââŒâââââââââââââââââââŒââââââââââââŒââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ€
â + â LLD â - â /opt/rocm/llvm/bin/ld.lld â
â + â Device Libraries â - â /home/simi/.julia/artifacts/5ad5ecb46e3c334821f54c1feecc6c152b7b6a45/amdgcn/bitcode â
â + â HIP â 6.1.40093 â /opt/rocm-6.1.3/lib/libamdhip64.so â
â + â rocBLAS â 4.1.2 â /opt/rocm-6.1.3/lib/librocblas.so â
â + â rocSOLVER â 3.25.0 â /opt/rocm-6.1.3/lib/librocsolver.so â
â + â rocALUTION â - â /opt/rocm-6.1.3/lib/librocalution.so â
â + â rocSPARSE â - â /opt/rocm-6.1.3/lib/librocsparse.so â
â + â rocRAND â 2.10.5 â /opt/rocm-6.1.3/lib/librocrand.so â
â + â rocFFT â 1.0.27 â /opt/rocm-6.1.3/lib/librocfft.so â
â + â MIOpen â 3.1.0 â /opt/rocm-6.1.3/lib/libMIOpen.so â
âââââââââââââŽâââââââââââââââââââŽââââââââââââŽââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
[ Info: AMDGPU devices
ââââââŹâââââââââââââââââââââââââŹâââââââââââŹââââââââââââŹâââââââââââââ
â Id â Name â GCN arch â Wavefront â Memory â
ââââââŒâââââââââââââââââââââââââŒâââââââââââŒââââââââââââŒâââââââââââââ€
â 1 â AMD Radeon RX 7900 XTX â gfx1100 â 32 â 23.938 GiB â
ââââââŽâââââââââââââââââââââââââŽâââââââââââŽââââââââââââŽâââââââââââââ
from waterlily.jl.
Great! And yes, that mismatch might have caused your original error. For that, you could submit an issue on AMDGPU.jl I guess. So, is this issue resolved now? :)
from waterlily.jl.
Related Issues (20)
- Test fail on MacOS in testing enviroment HOT 2
- Courant number (CFL) HOT 2
- Change default time in force integration HOT 3
- Circle example not working HOT 3
- Taylor Vortex file HOT 2
- Write VTK size/shape of field in writer HOT 2
- Can WaterLily.jl be used for 2D hydrodynamic modeling of real watersheds, similar to lisflood-fp, mike21, and anuga? HOT 2
- Sorry, my ignoramus questions about the Waterlily again HOT 1
- Simulating a tank for VIV HOT 1
- AD on GPU backend
- Noisy dependence of the body position HOT 10
- docs badge on `README.md` a broken link HOT 1
- Change CI to pass both CPU and GPU tests
- Internals flows in arbitrary shapes HOT 3
- questions about waterlily HOT 11
- Using DifferentiationInterface HOT 2
- Questions about STL files for simulation HOT 1
- Julia 1.10.4 vs 1.11.0 performance test
- Force Differences in WaterLily Version HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. đđđ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â€ïž Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from waterlily.jl.