inductiva / inductiva

Large scale simulations made simple.

Inductiva: a Python package for scaling simulations on the Cloud

Welcome to the official Python library for the Inductiva API version 0.5. The Inductiva API allows running a set of open-source physical simulators on the cloud, easily parallelizing simulations, each running on hundreds of CPU cores.

Inductiva simplifies the complexities of cloud resource management, and software configuration, offering a straightforward Python interface for running simulations on state-of-the-art hardware. This allows scientists and engineers to focus their time and energy on what matters: running simulations that solve real problems.

To find out more, check our documentation.

If you have any questions or suggestions about the API please open an issue on the inductiva’s API Client GitHub repo, or contact us via [email protected].

inductiva's Issues

The plots of the Dambreak scenario need to be uniform with parameter variation

In order to compare the flows of different fluid types we need to have the Dambreak plots being consistent across all of the parameters.

This can be simply done by setting all plots to have range [0., 1.] on each axis.

E.g.: two different types of fluids with the same position and dimension, but the plots have different z-axis range.

Remove code duplication in scenarios by introducing General scenario

The goal of this is to basically create a general scenario that serves as a basis for other more particular scenarios and as a intermediate level of the API.

As of now, creating new particular scenarios increases technical debt, since it requires a lot of code duplication.

See Maria's suggestion:

from inductive import fluids

fluid_properties= fluids.FluidProperty(viscosity, density)

fluid_block = fluids.FluidBlock(fluid_properties, fluid_velocity)

general_scenario = fluids.Scenario(fluid_block_list, rigid_bodies_list,...)

general_scenario.simulate()

Add low level API call for swash

Deploy SPH executers on multiple servers

Rather than having just one of the servers doing the computation.

Save stdout and stderr files to output directory

SplishSplash and DualSPH for the dam break scenario

As a user of the current dam break demo, I have the need to show investors that it is possible to seamlessly change the underlying simulator without changing anything else on my code, except the simulator parameter. This is a major selling point of the API because it allows us to show we are not stuck to a specific provider of simulation (and so we are not in a fragile business position that we depend on a single point to expose a certain functionality). I need to be able to show we can do this:

sim_output = scenario.simulate()

which runs with some default simulator (and which the user doesn't have to select but we will inform the user via a status message), but also:

sim_output = scenario.simulate(engine="splishsplash", device="gpu")

or

sim_output = scenario.simulate(engine="dualsphysics", device="cpu")

the feedback messages should show that we are running with these options, and the output should be syntactically the same so that the rest of the code does not change.

This is an urgent feature since it allows us to show a very important aspect of the API

Note: I would like this to be implemented by Luis+Begaiym with only distant supervision from Fabio.

Add viscosity solvers to SplishSplash

This is the first step to then pre-set fluids with different densities and requirements (issue #40).

This is low-hanging fruit and only requires changing the viscosity_solver to "Weiler_2018".

Allow more pre-set fluids to be simulated (in the DamBreakScenario)

Currently we have:

scenario = fluids.scenarios.DamBreak(fluid=fluids.WATER,
                                     fluid_dimensions=(.2, 1, 1),
                                     fluid_position=[0.0, 0.0, 0.0],
                                     resolution="medium",
                                     time_max=2)

Besides fluids.WATER, it would be important to add a few more fluid definition (if they already exist please add comment on that, or create an array with fluids and pass fluid=available_fluids[0/1/2] as parameter).

Ideally, we would have fluids that are very different from each other. Some suggestions (not sure if they are good ones):

1. olive oil (yes, we are thinking about a number of industries)
2. liquid propane/GPL
3. wood pulp / paper pulp (https://www.tappi.org/content/tag/sarg/t230.pdf)
4. typical sewage sludge (yes. that)
5. typical freshly made concrete (mixture of cement, water, sand)
6. hot glass (at the temperature you use for making bottles by injection)
7. other fluids that have massive industrial use / are transported at scale

API method to run a swash simulation remotely

Discuss the best way to check for the SciPy sparse matrix type

Discuss about a more effective way to provide a user with simulation feedback

Accelerate Low-Level Swash - #1 Customer Feedback

Customer feedback @hpenedones @sarmento:

After the FEUP meeting, me and @fabiocruz have discovered that what the researchers want is full control of the simulation.
And by that, I mean that they want to write their own input files, which gives them the ability to change everything, and then send them to us, via API, to obtain "faster" simulations.

A reason for this is to encounter a stable numerical simulation that afterwards can be iterated with some parameters to obtain an optimal configuration.

As of now, we almost have the low-level call for swash ready! This will be a nice step moving forward.
However, given that a single simulation of them takes 8h to run in 16 cores, we have to be able to run swash with multiple cores.

• Run swash with multiple cores.

This will be a first iteration of what we will present to them. As soon as it is ready, we show them again and start iterating on the product.

Add support for low-level DualSPH simulator

This is a requirement to then implement #42

"Visual" particle size remains constant, despite "simulation" particle size is changing (?)

It seems that the rendering of the particles using Matplotlib does not take into account the "particle size", so particles are always (visually) rendered with the same radius. We should consider automatically changing the radius of the rendered particles to reflect the dimensions of the simulated particles.

Note: Although matplotlib is not the best way to render SPH simulations, this is the tool we have now, so we should try to make it as good as possible.

API call to render objects (.obj files) for simulations

We may at some point provide functionality to generate meshes for standard objects via the API.

Provide intermediate feedback during simulation

We would have the backend sending progress information to the client side that could be displayed, for example, the number of time steps already simulated.
This might require some backend functionalities that we don't have now involving Redis.

Update a visualisation script to create movie from .nc files

Consolidate demos in Jupiter notebooks under GitHub

Use `isistance()` instead of comparing the type annotation for packing input

          I don't think it would be a problem to use `isinstance()` simply on pack (and not unpack), because we control the return types of the API, and hence the unpacking....

But let's keep it as is for now and open an issue for further discussion!

Originally posted by @fabiocruz in #2 (comment)

Implement FluidTank scenario

Import scenario implemented for wine tanks into the API

GPU submitted info wrongly provided by logs

This is the logs after I have tried to execute a scenario with GPU.
Since it was taking super long I accessed carbon and saw that the job was not being executed, because the GPU is being used by another different task.

The logs should reflect this, but they are like follows:

Low-level xBeach implementation

To gather more than one customer as of now, we can quickly add the low-level xBeach implementation to provide to other researchers at FEUP. This will bring a lot of issues but we will be building the product!

This will allow us to obtain extra feedback from more customers in what approach we should move forward.

Set boundary conditions for parameters in dam break scenario

Set boundary conditions for time (duration of the simulation) and resolution.

Portability to Windows clients

After testing a simulation on Colab, our FEUP users tried to test running things through Visual Studio Code with a simple python script.

He processed the installation with pip install .... our api package, which imports the various inductiva packages:
inductiva; inductiva-sph ; inductiva-data... etc

After this, he tried to run a simulation of swash with the low-level approach, which is fully embedded inside inductiva. However, he was not capable of running a swash simulation, because there was a python module that he could not install. The error followed:
ModuleNotFoundError: No module named 'resource'

This module only is compatible with the linux ecosystems and is being imported through the inductiva-sph package. It is used in the function run of splishsplash for the purpose of running a simulation locally. Therefore, this function is not used in the API at all, but is still imported to the client side and brings this problem with it.

There is a quick fix, which I already implemented by creating a branch win_version that does not use this module at all, thus there is no conflict when the user installs this branch instead. The purpose of this quick fix is to not break the feedback loop with users, while we fix this on the background, my decision was based on him not even using splishplash now.

This hides the true problem which we need to fix: should things that are not necessary for running simulations via API (or used at all on the API) be imported by the client? See #Resources for an inital part of the discussion and the quick fix.

Moreover, as we are implementing stuff we should be careful with portability issues of some packages (mostly hardware related).

Common parameters for DamBreak demo should "work"

The low resolution was too low... it generated just a handful of particles..

The high resolution was "too high" (at least in combination with duration=5 s -- which is not too long -- the simulation was running for more than 45 mins)...

Maybe the particle radius that corresponde to "low", "medium" and "high" need to be adjusted a bit.

As a reference, estimate the the particle radius to roughly match:

Resolution:
LOW : 1k particles
MEDIUM: 10k particles
HIGH: 100k particles
VERY_HIGH: 1M particles

(rough number of particles -- it can be slightly different)

As a reference, the current HIGH setting led to about 24M particles...

"Waiting for resources: X jobs ahead in queue. Please hold..." message

As we put the notebook in the hands of users and potential investors, we want to make sure the we have capacity to respond to their requests, or if we are running out of capacity, we explain how many jobs are ahead to be served before we can process that user's job.

Develop a mechanism to give feedback of the size of the queue when a job is submitted.

This is important for the user experience but also for investors to see that there are jobs being submitted.

Add "data operations" on resulting simulation data [notebook demo]

Currently, the notebook shows a video as the result of the simulation. That is great to help people understand what is going on. However, we also need to show we can get access to the simulation data and do something with it.

I would like to had a piece of code after the video that retrieves the simulation data (and iterates over it) to find the time when the water hits the back wall for the first time.

This will show the user how to retrieve that data, and will illustrate the resulting data structures for SPH.

Change the color of the fluid in the visualization depending on the fluid being simulated.

Introduce JAXFLUIDS explosions to the API

An important framework to simulate compressible fluids, in particular explosions, is provided by the library JAXFLUIDS.

This provides a differentiable simulator to integrate in DL experiments. For us, the important quest is to test it in best
possible hardwares at Inductiva. With this in mind, setting it up in API, in a low-level implementation, provides a simple framework
to test and show to the Clients.

Create a skeleton for other to add more simulators to the API

As of now, to integrate a new simulator in the API at this point we need to pass most of the work to Luis and Begayim. Most of the team is not aware how to integrate in an appropriate way these simulators.

The goal is to add documentation and a skeleton so that anyone can start from there to add a simulator to the API and then you only need to review and improve some of the implementation details.

-> Add documentation and necessary scripts so that any member of the team can start adding simulators to the API.

Fluid dimensions validation is missing

Playing with the API, I have noticed that we can insert non-positive values to the fluid_dimensions which does not make sense.

• Fluid dimensions need to be positive and bigger than the particle radius.

Argument type used to identify that directory needs to be uploaded

"Right now we rely on the type annotation to know that the argument is a directory and that we need to include that directory in the zip that we send to the API. So just removing this doesn't solve the issue."

A possible solution would be to use pathlib.Path as the type, and then checking if the path points to a directory. Wdyt?

Originally posted by @luispcunha in #23 (comment)

Update existing executers to use the `BaseExecuter` class

Add boolean to characterise newtonian nature of fluid types

Some fluids can be newtonian and non-newtonian. Then solvers need to act different and may have different SPH properties.

Move simulation-related parameters to simulate()

Currently we have:

scenario = fluids.scenarios.DamBreak(fluid=fluids.WATER,
                                     fluid_dimensions=(.2, 1, 1),
                                     fluid_position=[0.0, 0.0, 0.0],
                                     resolution="high",
                                     time_max=5)

sim_output = scenario.simulate()

however, the parameters: resolution="high", time_max=5 are not properties of the scenario, but of the simulation.

Ideally we would have something like:

scenario = fluids.scenarios.DamBreak(fluid=fluids.WATER,
                                     fluid_dimensions=(.2, 1, 1),
                                     fluid_position=[0.0, 0.0, 0.0])

sim_output = scenario.simulate(resolution="high", time_max=5)

Add low level call to GROMACS

from inductiva import molecules

scenario = molecules.Gromacs(
     path_to_my_local_config_file,
     directory_with_extra_artifacts,
     command_line_arguments)

sim_output = scenario.simulate()

Add support for lower level call of splishsplash

Currently we have a "pre-canned" use case for the DamBreak:

scenario = fluids.scenarios.DamBreak(
                                     fluid=fluids.WATER,
                                     fluid_dimensions=(.2, 1, 1),
                                     fluid_position=[0.0, 0.0, 0.0])

sim_output = scenario.simulate()

This use case happens to be implemented in splishsplash. We would like to have a lower-level call that takes as input a config file (and a directory with artefacts) and sends everything to be run on the server side.

We would like to have an even lowerlevel call that allows the user to run splishsplash one a config file + artifacts that he has already in his computer. This give the user all the flexibility, which is important for some power users. E.g.

scenario = fluids.SplishSplash(
     path_to_my_local_config_file,
     directory_with_extra_artifacts)

sim_output = scenario.simulate()

For now, we can test this with the dam-break scenario, meaning that we can submit whatever file we are already generating using the dambreak use case.

FPS of video and simulation not matching [notebook]

In my experiments I am generating 2 seconds of simulation data.
I believe we are generating 120 ou 121 frames of simulated data but I am not sure because I have no feedback. I do remember that talking with Fabio about the simulation step being 1/60s, so that is way I am saying we should have about 120 frames of data.

However, when I watch the video, the video takes 12 seconds to run, which means that it is being shown 6 times slower than real speed. As a user, I was not expecting this. The default mode should be "real time". So this raises a number of question that I think we need to discuss. And there are multiple implications. For example, if we go real time on the video, and the video is a standard 12 fps (which is reasonable for monitors), then we only need to render 1 every 5 frames of data, which allows us to render the videos much quicker and improve the experimentation speed.

As a user, I think I need way more control on how the video is generated, and have more sensible defaults (i.e. I expect "real time" first).

How to pass simulator-specific arguments - namedTuple?

GPU support for SplishSplash

Currently we can only run on CPU, but it would be great to be able to use our NVIDIA GPUs.

Faster rendering: solutions?

Is it possible to generate videos faster on the client side? What are the bottlenecks?

If not, should the videos be generated on the server side, and benefit from much better hardware?

Let's first identify the current constraints, and come up with proposals. We can use this issue to discuss and document the options here, before implementing.

Are we generating the video twice?

Currently, when I run

sim_output = scenario.simulate()

I see a progress bar:

100%|██████████| 121/121 [00:01<00:00, 69.95it/s]

which is happening remotely (I guess) and shows about 70 iterations per second.

But then when i run the command to generate the video:

video = sim_output.render()

I get another progress bar for what seems to be the same thing but running more slowly:

100%|██████████| 121/121 [00:27<00:00,  4.45it/s]

This suggests -- but I don't know for sure because we still have no feedback -- that we are actually running video generation twice.

If so, should we add a parameter to simulate() and just download it?

sim_output = scenario.simulate(render_video=True)
video_html = sim_output.get_video_html_snippet()

Then the default behavior is not to generate the video (seems reasonable, because the user may actually not want to generate videos, just do some calculation with resulting tensors) but you can do it if you explicitly ask so.

Support for running simulations in parallel

Allow any user or create a specific user for pip install

Currently, pip-install uses Fabio's user name to clone the packages.

!pip install --no-cache git+https://fabiocruz:github_pat_11ACYCYIA0LwIqELZJ[e-mail]/inductiva/inductiva

We need to either a create a user for this or we need to open the repo for everyone.

Add "transparency" property to fluid description for more realistic renderings.

Inductiva fluids already have a color property (great!) but there seems to be missing a transparency property. Adding this property would improve the visual aspect of the animations.

Add wall in dam break scenario

Currently we have:

scenario = fluids.scenarios.DamBreak(fluid=fluids.WATER,
                                     fluid_dimensions=(.2, 1, 1),
                                     fluid_position=[0.0, 0.0, 0.0],
                                     resolution="medium",
                                     time_max=2)

sim_output = scenario.simulate()

which generates a simulation inside a box, unrestricted. Now we want to add a wall so that we block part of the fluid. We will start with a simple rectangular wall, where we define only the width and the height of the wall, and position from the bottom of the simulation box.

scenario = ....

scenario.add_rectangular_wall(height=0.3, width=0.05, distance_to_back=0.1)

sim_output = scenario.simulate()

This is a first step to creating more sophisticated walls, either by having different type of walls, or by defining a profile for extrusion (e.g. a line in .obj).

Infinite height in dam break scenario

Currently, the Dam Break scenario seems to have a maximum height, so particles easily hit the ceiling. This introduces an unwanted distortion to the scenario that we don't need to have, and it raises questions from the point of view of the user that we want to avoid. Change the height of the simulation box to a value that is much higher, while still only showing in the simulation only a region that is just 1 to 1.5 the "standard" height.

Requirements for generating Fluid Cube dataset

One possible use case of our API is to generate datasets of SPH simulations. This way, since I generated the Fluid cube dataset, we thought it might be a good test for the API to generate a dataset with the same properties and start building an example of how to use the Inductiva package for this use case.

The original script for generating the dataset is here.

As a first requirement, for generating the dataset we need to be able to set all the variables needed to replicate the conditions in that script. In my analysis I found the following properties missing from the API DamBreak scenario:

• Initial velocity of the fluid block: this is a parameter that is varied on the dataset and it is not possible yet to set a fluid velocity on the API simulations. Not sure if this fits a DamBreak scenario, a dam break usually doesn't have initial velocity, so maybe you want to have another type of more general scenario to add arbitrary fluid blocks. However, this is a key feature of the dataset.
• Particle radius: I noticed you allow for a low, medium, and high resolution but to replicate the dataset and generate new datasets it would be useful to set this parameter.
• Time step: allow setting the time step.
• CFL: the current CFL condition is set by default to "cfl" to generate the dataset we need to set this to "no".
• reordering: Need to disable the reordering of the particles during the simulation because the particles need to be trackable.
• pass arbitrary viscosities using the Inductiva package

Discuss where to store outputs and whether `output_dir` should remain as an argument of `inductiva.init()`

          Reminder to create an issue to discuss where to store outputs and whether this `output_dir` should remain as an argument of `inductiva.init()`.

Originally posted by @fabiocruz in #2 (comment)

Use the `npz` format instead

          Should we use the `npz` format instead? It's true we already zip the inputs after this, but this would zip each array in an optimal way (handled by numpy and not us).

Originally posted by @fabiocruz in #2 (comment)