empymod models the EM Green's function for electric and magnetic sources and receivers, and can, as such, model any EM method and any survey layout. However, the implementation can sometimes be tricky. It would therefore be good to add some easy high-level routines to model some of them, e.g. TEM.

One such feature that is possible, but could be made much easier, is arbitrary waveforms. Currently the high-level time-domain routines only contain impulse and step-on and -off functions. However, it is not too difficult to allow any arbitrary waveform.

Alternatively, if not implemented properly, than we should at least provide some detailed examples.

Move the Example-Notebooks repo into a Sphinx Gallery.

Development of v2 - overview and progress

These MUST be done; then create a release candidate:

• #13 Change bad names (#84)
• #14 Abstraction, sort of (#85)
• #15 Change htarg/ftarg to dicts (#81)
• #21 Replace numexpr by numba (#77, #79)
• #73 Ensure inputs are not altered (#83)
• #76 Improve EMArray (#78)
• Drop Python 3.5 support; start using f-strings ; remove backwards compatibility (#72, #80)
• Check defaults => change to fast ones, not most precise ones (e.g., {'pts_per_dec': -1})

With a release candidate in place, tackle these:

• Check documentation to reflect changes.
• Check docstrings and inline comments to reflect changes.
• Adjust and verify benchmarks empymod-asv.
• Adjust and verify examples.

The addons empyscripts are by now in pretty good shape:

• Travis-CI integration
• Coveralls integration
• Documentation

Originally, I created empyscripts because the code wasn't as thoroughly tested and not as stable as empymod, so I did not want to mix the two. However, given the current state of empyscripts it would probably be easier to have it in empymod.addons (or a better name?), so users don't have to install two packages. After all, even the two together have still a pretty small footprint.

Things to think about:

• Name? (empymod.addons, empymod.scripts, ...?)
• Move the current code of empymod into empymod.core?
• Make any additional requirement from empyscripts a soft requirement? (E.g. matplotlib).
• Move printinfo into empymod.utils or into empymod.addons?

Decide soon! If it is to be moved it would be good to move it before article-fdesign and csem-ziolkowski-and-slob are released.

The wavenumber-domain calculation is exact. However, in the land case where source and receiver are at the surface some numerical artefacts make early-time responses (very high frequencies) useless.

Create a fix which checks if source and receiver are at the interface between air and subsurface, and eliminate the critical part in the wavenumber domain.

Related to notebooks

• Section_5-4-3.ipynb,
• Section_5-4-4.ipynb, and
• Section_5-4-5.ipynb

in the repo empymod/csem-ziolkowski-and-slob.

Theoretical correction exists, it just has to be implemented correctly.

Improve docs, namely:

• Include README.md, CHANGELOG.md, THANKS.md (change to rst?) into sphinx-doc.
• If change to rst: Check setup.py (check PyPi, Conda).
• Generally simplify, improve, make nicer/cleaner; check all.
• Avoid as much as possible duplication in empymod.github.io; README.md; and init.py.

Maybe in parallel with emsig/empyscripts#2.

Change empymod from positive z downwards to positive z upwards, to be compatible with emg3d and most 'modern' geo-software and display tools.

Backwards incompatible, something for version 2.0.

See comment below, it is already possible. We just have to document it better:

• Manual
• Docstrings
• Example notebook

I received an email today, stating:

The question I have is related to induced polarization. A popular representation to 'describe' this phenomenon is through Cole-Cole parameters. Am I right in that empymod cannot handle this particular implementation? I have been looking at the documentation and examples, but apparently missed something (I hope).

This issue is hence to create a routine to calculate IP with Cole-Cole-parameters.

A list of things to be changed in the backwards-incompatible version v2.0.0.

Do

• transform fct names
  • transform.get_spline_values -> transform.get_dlf_points (#81)
  • transform.fht -> transform.hankel_dlf
  • transform.hqwe -> transform.hankel_qwe
  • transform.hquad -> transform.hankel_quad
  • transform.ffht -> transform.fourier_dlf
  • transform.fqwe -> transform.fourier_qwe
  • transform.fftlog -> transform.fourier_fftlog
  • transform.fft -> transform.fourier_fft
  • transform.fhti -> transform.get_fftlog_input
• Parameter names (more pythonic / PEP8)
  • PEP8: re-name factAng -> fact_ang
• Other changes
  • Possibility to only provide {e;m}permH, and {e;m}permV will be copied (#81)

Don't

Other parameter names (more pythonic PEP8); not worth the troubles to change.

• More explicit parameters, e.g., source, receiver, resistivity
• {e;m}perm{H;V}=> {e;m}perm_{h;v} (externally)
• Split-up freqtime to frequency and time
• {z}eta{H;V} -> {z}eta_{h;v}
• There are so many mixed-case variables; just leave them (e.g., PTM, PTE, PJ{0;1}, W{u;d}, P{u;d}, fEM, tEM, and dozens more).

Since commit 2d3171a there is a name-clash in the main namespace empymod.? of model.wavenumber and kernel.wavenumber. As the kernel is loaded after the model, kernel.wavenumber is available as empymod.wavenumber, while model.wavenumber is not.

Decision is needed:

• Rename one of them?
• Load one of them under an alias?
• Rethink the whole name-space?

Setting xdirect=False in bipole and dipole (calculating the direct field in the wavenumber-frequency domain) is much faster than having it at the default xdirect=True. Additionally, it is more consistent, as everything is calculated in the same domain.

Before change check if there are any issues setting xdirect=False by default.

The first test in fdesign keeps` failing randomly, restarting resolves it. FIX THIS TEST.

Travis fails for Python 3.6 and 3.7 (not for 3.5). Something to do with numpy I think; pickles, numpy format, and npyio.

After #7. Add modelling routine for TEMs (TDEMs):

• Central loop
• Loop-loop (already possible, model.loop)
• Coincident loop

For the coincident loop there is the issue of zero offset.

Add more examples:

• Figures 4.7 and 4.8 of Ward and Hohmann (added to Magnetic loop sources and magnetic receivers)
• Add examples for the following layouts:
  • central loop (implemented in TEM: ABEM WalkTEM)
  • loop loop
  • concentric loop
• A examples for ramp and arbitrary waveform; pulses; filtering.

• Automate remove badges from readme for PyPi.
• Automate versioning.
• Make Travis deploy to PyPi.
• Move to conda-forge.

After this step, tagging is the only thing required to release a version.

Since empymod v1.4.2 (Sept. 2017) the citations in the docs are broken; an additional 0 is added to the label, for which reason all subsequent citations in the code are "undefined".

Compare empymod.readthedocs.io/en/v1.4.2 to empymod.readthedocs.io/en/v1.4.1. There is no docs-related change between the two versions, yet the citations are broken in v1.4.2 onwards.

It seems that this is related to a change in numpydoc from v0.6 to v0.7.0, see numpy/numpydoc#114 and numpy/numpydoc#136.

Wait for this issue to be resolved and verify that it fixes the problem with the empymod-docs.

For now, I set numpydoc==0.6.0 in requirements.txt.

There are some repetitions in transform if looped over frequencies or offsets that could be avoided. In the case of the lagged convolution Hankel DLF this can make as much as 10 % of the runtime (but then, in the case of the lagged convolution Hankel DLF, the runtime is VERY fast, so overall gain is probably rather small in absolute numbers and might not be worth the effort).

• Move lambd-calculation outside of empymod.transform, to avoid repetition if looped over frequencies.
• Pass factAng instead of angle to empymod.transform, to avoid repetition if looped over frequencies or offsets.

Note: This is an issue I received per email. I reproduce it here so others with the same issue might find the answer.

Hi,

I have just started using python to run your empymod code because I want to run a simple GPR 2-layer model. I have used anaconda and set up the empymod from the prisae channel. I have numpy and scipy installed. However I get an error on the plotting in jupyter notebooks;

FileNotFoundError: [Errno 2] No such file or directory: 'latex': 'latex'.

Can you give me any advice on how to find the 'latex' folder. I have done some searching on the web, but I have got nowhere.

Thank you

The Travis-CI test fails randomly when using CHAN=conda-forge (currently the 5th out of the 6 tests environments), and has to be restarted 1-2 times in order to succeed. The problem is the second test in empymod/scripts/fdesign.py.

Fix that.

In model.fem, zeroes are returned for source-receiver combinations ab=36 and ab=63. The routines model.bipole and model.dipole make use of model.fem. However, model.analytical, model.wavenumber, and model.gpr do NOT use model.fem, and therefore fail if they are called with ab=36/63.

e19f406 fixed this for model.wavenumber, but it is still a bug in model.analyticaland model.gpr.

Correction 1: model.gpr uses internally model.dipole, so it actually goes through model.fem.
Correction 2: Diffusive half-space is only implemented for el. src and el. rec. So we only have to take care of empymod.analytical(solution='fs').

These really should be dicts and not lists, with afterwards fragile order-depending stuff like filt=htarg[0] etc.

In empymod.transform.get_spline_values, I use log instead of log10. So pts_per_dec is actually not per decade, but per "e".

Either adjust docs or code.

In empymod.bipole, when src or rec is formatted as [x0, x1, y0, y1, z0, z1] with multiple bipoles with different azimuths, the fields are computed as if all bipoles have the same orientation as the first in the list. For example, for two orthogonal sources:

import numpy as np
from empymod import bipole

xx = np.linspace(-100,100,40)
yy = np.linspace(-100,100,40)
X,Y = np.meshgrid(xx,yy)
xs = X.flatten()
ys = Y.flatten()

epm_args = {
    'src':[[0,20],[20,20],[0,0],[0,20],1e-2,1e-2],
    'rec':[xs,xs+10,ys,ys,1e-3,1e-3],
    'depth':[0],
    'res':[1e20,1],
    'freqtime':1,
    'srcpts':15,
    'verb':0,
    'epermH':[0,1],
    'epermV':[0,1]
}

bex = bipole(**epm_args)
np.unique(bex[:,1])

All entries of bex[:,1] are 0+0j. Similarly, for two orthogonal receivers,

epm_args = {
    'src':[0,20,0,0,1e-2,1e-2],
    'rec':[[30,30],[35,30],[30,30],[30,35],1e-3,1e-3],
    'depth':[0],
    'res':[1e20,1],
    'freqtime':1,
    'srcpts':15,
    'verb':0,
    'epermH':[0,1],
    'epermV':[0,1]
}
bex = bipole(**epm_args)
bex[1]

Output: 0j.

Including automatic link checking when docs are built.

Add additional parameters:

• unwrap={True;False}
• deg={True;False}

Inversion capabilities

=> Inversion Milestone

• Analytical gradient
• Jacobian
• A Model-Class that does checks inputs only when necessary (related to #180)
  • Class can do everything that dipole/bipole/loop/analytical does
  • Checks only done when necessary
  • Logging!
  • ...
• Examples using external tools (pyGIMLi, SimPEG, SciPi, wbi, ...)
• Bring over noise functions from emg3d
• Wrappers to make integration with external tools (pyGIMLi, SimPEG) easy

Attempts in the wild:

• SimPEG: https://github.com/OpenGeophysics/scratchpad
• SimPEG/SciPy: https://github.com/prisae/dev-pygimli-emg3d/blob/tmp/empymod-SciPy-SimPEG.ipynb
• pyGIMLi: https://github.com/gimli-org/transform2021/blob/main/6_Inversion_with_any_forward_operator.ipynb
• pyGIMLi: https://github.com/prisae/dev-pygimli-emg3d/blob/tmp/pyGIMLi-empymod.ipynb
• wbi: https://github.com/WouterDls/1D-wavelet-based-inversion/blob/main/examples/scripts/example_empymod.py
• Work of @mariacarrizo (add links)
• Work of @aignerlukas (add links)

• Include all CI-things in MAINTENANCE.rst
  • travis
    • pytest
    • flake8
    • linkcheck
    • coveralls
  • codacy
  • rtd
  • zenodo
  • asv [not yet automatic]
  • sphinx-gallery [not yet automatic]
  • deploy
    • pypi [not yet automatic]
    • conda [not yet automatic]
• Include MAINTENANCE.rst in documentation (just like README.rst, CHANGELOG.rst)

There are currently two branches which try to implement numba:

• https://github.com/empymod/empymod/tree/numba
• https://github.com/empymod/empymod/tree/numba-newtry

Roadmap:

• Properly implement the four kernel functions.
  • wavenumber()
  • greenfct()
  • reflections()
  • fields()-
• Further Benchmarks and Profiling
  • Test with parallel=True and nb.prange
  • https://github.com/empymod/empymod-examples/blob/master/6b_Profiling.ipynb
  • https://github.com/empymod/empymod-examples/blob/master/6a_Benchmark.ipynb
  • => Decide if this is enough, or if also parts of transform.? have to be jitted.
  • Adjusts pytest for numba
  • Ensure documentation on RTFD for numba builds properly
• Remove completely numexpr
  • from code
  • from documentation etc
  • requirements
  • examples
  • README
• In other repos:
  • Adjust website (only once v2.0 is released)
  • Adjust benchmarks (empymod-asv)

Below was the original commit from 2018.

Currently 13 of the heaviest computations are implemented twice, once with NumPy, once as strings for numexpr (switch opt=None or opt='parallel'). These are all in empymod.kernel, and are

• 1x sqrt of complex matrix of size (nfreq, noff, nlay, nwvnr)
• 11x exp of complex matrix of size (nfreq, noff, nwvnr)
• 1x division of complex matrices of size (nfreq, noff, nwvnr)

Maybe these two implementation should be replaced by one single implementation using numba. See the notebooks

• Complex-Sqrt_Numpy-Numexpr-Numba-Cython.ipynb
• Sqrt_Numpy-Numba.ipynb
• Exp_Numpy-Numba.ipynb

in https://github.com/prisae/tmp-share.

It looks like a speed-up of up to 70% could be achieved. At the cost of readability. But then, given the first list, 3 numba-functions might be enough; sqrt, exp, and the division.

Something to think about.

Create main classes

• survey,
• model, and
• settings,

where settings might be part of survey. Have a look at SimPEG and a chat with the SimPEG folks to discuss this.

kernel.wavenumber returns three responses:

• PJ0: angle-independent part with a 0th order Bessel function,
• PJ1: angle-dependent part with a 1st order Bessel function (if ab=33 angle-independent),
• PJ0b: angle-dependent part with a 0th order Bessel function.

Depending on which response is calculated, some of these might be zero:

At the moment PJ?'s which are pure 0s are treated as if they had information. Proper handling of those could speed up the calculation, as less Hankel transforms would have to be carried out.

Currently the test suites fails if run with numexpr that was not built against VML (using, for instance, anaconda-forge instead of anaconda).

One reason is that numexpr returns NaN for "exp(-infty)", whereas numpy returns 0 for np.exp(-np.infty). This causes certain tests to fail, which compare the normal versus the numexpr version.

There might be, very likely, other issues.

The test suite should be able to run successfully against both.

Up to empymod v1.3.0 the repo contained two more directories, notebooks and publications, which where moved into their own repos afterwards (example-notebooks and publications was split into article-geo2017 and article-tle2017).

A current check-out of the empymod-repo is almost 80 GB. I suspect that probably about 80 % of this weight is due to these two old directories (notebooks and figures for the publications). The other 20 % are mostly due to data in the test-directory.

It could be a good thing to get rid of that ballast. But it means messing with the git history, so it has to be done very carefully.

empymod changes some input parameters, e.g., src and rec. Example:

In [1]: from empymod import dipole 
In [2]: rec = [5000, 0, 200] 
In [3]: rec                                                                     
Out[3]: [5000, 0, 200]
In [4]: EMfield = dipole([0, 0, 200], rec, [], 1, 1, verb=0)              
In [5]: rec                                                                     
Out[5]: [array([5000.]), array([0.]), array([200.])]

Ensure that this does not happen. As this will break some code that assumed this 'behaviour', it should be put into the v2.0.0 release.

Hi Dieter Werthmüller,
Very nice work. When I run a simple example given in model.py:

import numpy as np
from empymod import dipole
src = [0, 0, 100]
rec = [np.arange(1, 11)*500, np.zeros(10), 200]
depth = [0, 300, 1000, 1050]
res = [1e20, .3, 1, 50, 1]
EMfield = dipole(src, rec, depth, res, freqtime=1, verb=0)
print(EMfield)

The results are

[  4.58713929e-11 -1.71360411e-10j  -6.97266213e-12 -1.00555912e-12j
   2.57717900e-14 +4.99252420e-13j   4.36445152e-14 -1.20355003e-14j
  -2.23112374e-15 -4.12011680e-15j  -3.93535170e-16 +3.51191751e-16j
   5.16764963e-17 +3.58827958e-17j   2.82917050e-18 -7.29853500e-18j
  -9.97105343e-19 -1.31771731e-19j   1.42759606e-20 +1.31775378e-19j]

, which are not the same with the results

[  1.68809346e-10 -3.08303130e-10j  -8.77189179e-12 -3.76920235e-11j
  -3.46654704e-12 -4.87133683e-12j  -3.60159726e-13 -1.12434417e-12j
    1.87807271e-13 -6.21669759e-13j   1.97200208e-13 -4.38210489e-13j
    1.44134842e-13 -3.17505260e-13j   9.92770406e-14 -2.33950871e-13j
    6.75287598e-14 -1.74922886e-13j   4.62724887e-14 -1.32266600e-13j]

given in model.py. Did I get something wrong? Many thanks.

FYI, the OS is windows 7 (64 bit).

Replace np.int, np.float, and np.complex with either the pure Python equivalents int, float, complex, or with the numpy versions np.int_, np.float_, and np.complex_.

numpy/numpy#3998 (comment)

They are likely going to be deprecated and removed in the future: numpy/numpy#14882

There is a bug in FFTLog for certain parameters:

MWE:

import empymod
import numpy as np

empymod.dipole(
    src=[0, 0, 0], rec=[5000, 0, 0], depth=[], res=1,
    freqtime=np.logspace(-2, 2, 201),
    signal=0, ft='fftlog', ftarg=[5, [-2, 1]]
)

Error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-88dbe35cac4d> in <module>
      3 
      4 empymod.dipole([0, 0, 0], [5000, 0, 0], [], 1, np.logspace(-2, 2, 201),
----> 5                signal=0, ft='fftlog', ftarg=[5, [-2, 1]])

~/anaconda3/lib/python3.7/site-packages/empymod/model.py in dipole(src, rec, depth, res, freqtime, signal, ab, aniso, epermH, epermV, mpermH, mpermV, xdirect, ht, htarg, ft, ftarg, opt, loop, verb)
    947     # Do f->t transform if required
    948     if signal is not None:
--> 949         EM, conv = tem(EM, off, freq, time, signal, ft, ftarg)
    950 
    951         # In case of QWE/QUAD, print Warning if not converged

~/anaconda3/lib/python3.7/site-packages/empymod/model.py in tem(fEM, off, freq, time, signal, ft, ftarg, conv)
   1587     tEM = np.zeros((time.size, off.size))
   1588     for i in range(off.size):
-> 1589         out = getattr(transform, ft)(fEM[:, i]*fact, time, freq, ftarg)
   1590         tEM[:, i] += out[0]
   1591         conv *= out[1]

~/anaconda3/lib/python3.7/site-packages/empymod/transform.py in fftlog(fEM, time, freq, ftarg)
    723         ar = a[2*m-1]
    724         ai = a[2*m]
--> 725         a[2*m-1] = ar*argcos[:-1] - ai*argsin[:-1]
    726         a[2*m] = ar*argsin[:-1] + ai*argcos[:-1]
    727         # problematical last element, for even n

ValueError: operands could not be broadcast together with shapes (17,) (16,) 

Create a comparison example with gprMax, to

• verify correctness of empymod.model.gpr,
• get an idea how they compare in terms of speed.

As comparison could serve these examples:

• https://github.com/empymod/article-geo2017/blob/master/notebooks/gpr-figure.ipynb
• https://github.com/empymod/article-fdesign/blob/master/notebooks/09-Figure_GPR.ipynb

Hi @prisae, I really like the idea of performing hankel transformation at once rather than looping over for frequency. So, for the time to frequency transform I want to do the same procedure, particularly when I am computing sensitivity. Hence, the input frequency response can have size (n_frequency x n_layer).

Following snippet of the code emulates the situation that I want to perform, but that outputs errors.
Have you tried this?

from empymod import filters, transform, utils
import numpy as np

fftfilt = filters.key_81_CosSin_2009()
time = np.logspace(-5, -2, 21)
n_layer = 19
time_out, freq, ft, ftarg = utils.check_time(time, 0, 'sin', {'pts_per_dec':2, 'fftfilt': fftfilt}, 0)
a =1e2
fEM =1j*freq*2*np.pi/(a + 1j*freq*2*np.pi )
signal = fEM/(2j*np.pi*freq)
resp, _ = transform.ffht(np.tile(signal.reshape([-1, 1]), (1, n_layer)), time, freq, ftarg) 
resp *= 2/np.pi

analytical fails for solution='dhs', solution='dfs', solution='dsplit', if at the same time several offsets AND several frequencies. Fix it! It works fine for solution='fs'`.

At the moment, scripts.tmtemod only works for ab=11. Expand it to work for all electric sources and receiver (and maybe even for rotated sources and receivers).

For electric sources and receivers it should be straight forward. Expanding it to magnetic sources and receivers is also possible, but it is not a task of pure programming, as it involves some maths beforehand to calculate the required non-physical contributions.

https://empymod.readthedocs.io/en/stable/addons.html#tmtemod-calculate-up-and-down-going-tm-and-te-modes

Hi,
is it possible to specify the strenght of analytical dipole source when using

import empymod as epm
model = {'res': 1 / 3.2, # Resistivity
'freqtime': 1, #
'signal': None, # Frequency domain
'solution': 'dhs', # Diffusive half-space solution
'verb': 1} # Verbosity

field[:] = epm.analytical(ab=25, **model)

Or is it set internally constant to 1A?

Abstract the whole transformation stuff. This probably includes stuff from:

• empymod.transform
• empymod.filters
• empymod.scripts.fdesign
• emg3d.utils.Fourier

This should work for (k-f <=> x-f; x-f <=> x-t; x-s <=> x-t)

• DLF (Hankel and Fourier)
• FFTLog (Hankel and Fourier)
• QWE (Hankel and Fourier)
• QUAD (Hankel)
• FFT (Fourier)

Base class:

• input domain
• output domain
• method
• arguments
• optionally interpolation

The derivation of empymod assumes unit dipole sources and receivers. If the magnetic source or receiver are, instead, loops, then they change by a factor j \omega \mu (per loop). This is currently not implemented.

The frequency-domain result can easily be multiplied by that factor afterwards. However, for time-domain results this is not possible.

Create a new loop-routine that allows to calculate for loops`.

Thanks @ruboerner for reporting it and starting a PR.

PR related to it: #45

Example notebook dealing with it: https://github.com/empymod/empymod-examples/blob/master/5b_TEM.ipynb

A recent conversation with @sgkang from SimPEG reminded me of some chaotic things within empymod, whose roots lie in the way the code grew over time, and also how I learned over time. For instance, the original DLF functions are called fht (Fast Hankel Transform), as only the Hankel transform was considered, time-domain got added later. And over time different possibilities were added for Hankel DLF and Fourier DLF, the most recent is Standard Fourier DLF which was only added with empymod v1.5.2.

DLF implementation and their different optimisations

The methods that are possible with empymod for the Hankel and Fourier DLF are:

• Standard DLF (No optimisation)
• Lagged Convolution DLF (interpolation in output domain); spacing defined by filter
• Splined DLF (interpolation in input domain), spacing defined by pts_per_dec.

Status quo

At the moment, this is differently implemented for Hankel and for Fourier:

Hankel DLF

• standard: pts_per_dec=None
• lagged: pts_per_dec=None, opt='spline'
• splined: pts_per_dec>0, opt='spline'

Fourier DLF

• standard: pts_per_dec<1
• lagged: pts_per_dec=None
• splined: pts_per_dec>0

Idea forward

I think a good implementation would be to have for both, Hankel and Fourier:

• standard: pts_per_dec=0
• lagged: pts_per_dec<0
• splined: pts_per_dec>0

Therefore only having one parameter, and getting rid of opt='spline'.

This would mainly affect empymod.transform, and implementation is relatively easy. Functionality in empymod.utils could be extended to keep it backwards-compatible for all empymod.model-routines (including opt='spline'). However, the changes would be backwards incompatible for empymod.transform-functions. So it has to be done carefully to not break code which depend on these routines.

In 30c5f23 versioning was handed over to setuptools_scm. Unfortunately, setuptools_scm handles what is included in a build, it includes by default everything under version control, which is definitely not, what we want. MANIFEST can be used to exclude stuff, so we have to update that.