External potential operator in plane wave basis non-Hermitian for even grids about openfermion

[max-radin]

Sure I can help with this. Is it sufficient just to change the behavior of the momentum_vector function?

[babbush]

It has not been resolved. That would be awesome @conta877 !

[gkc1000]

No problem, feel free to chat more. Even grids are odd.

…

On Sun, Mar 25, 2018 at 8:19 PM, Jarrod ***@***.***> wrote: Thanks for the clarifications Garnet. I’m confident that the implementation can be made hermitian as you say; I guess the question is whether doing so with our other approximations immediately implies the diagonal property in the dual basis for any choice of integer grid. I’m only going off what people have written here(as opposed to the code or my own derivations) so far as I was planning to look into this this week as part of something else and wanted to get a feel for the current status, but your points will definitely be helpful in that investigation. > On Mar 25, 2018, at 7:58 PM, gkc1000 ***@***.***> wrote: > > Since my mailbox is filling up ... > > I'm not completely sure what is implemented inside OpenFermion, but: > > Hermiticity is obviously a property of the operator i.e. so we can choose an arbitrary set of basis functions and <m|H|n> = <n|H|m>*. So any possibility of observing non-Hermiticity can only arise from an approximation in computing the matrix element <m|H|n> (in relation to the earlier questions with max-radin). > > The approximation that corresponds to the discrete-variable representation + dualing in the PRX paper can be interpreted for the nuclear potential V as > > \sum_{i} <m|R_i><R_i|V|R_i><R_i|n> > > where <m|R_i> is taken to be evaluation of the function at grid-point R_i, and the grid points are evenly spaced in the box. This is Hermitian. > > if I look inside the plane_wave_hamiltonian.py, it looks like the above is implemented in plane-wave space, i.e. one is working with the elements V(G_i) such that > V(R_i) = DFT[ V(G_i) ] > > To ensure Hermiticity, one has to ensure that V(R_i). This implies that V(G_i) = V(-G_i)* where G_i is "aliased" as in a discrete FT. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub, or mute the thread. > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvWxIt7ey3MBiWHTVBSyM_4MW_bqrRjks5tiF49gaJpZM4SJ2s6> .

[babbush]

Thanks for pointing this out. The issue can't occur in dual without also occurring in plane waves since one is the Fourier transform of the other. Based on the equations in 1706.00023 there shouldn't be a problem. Thus, there is likely some bug in our implementation.

[max-radin]

Ok! Just to be clear, I think the behavior is consistent between the plane wave basis and dual plane wave basis – both give non-Hermitian operators with even grids in the current implementation.

[babbush]

I've spent a while looking into this. The problem has to do with the way we set the momentum and real space grids when there are an even number of grid points. The problem is a fairly fundamental one which has to do with even versus odd grids. For an odd grid, we can take momenta values [-2, 1, 0, 1, 2]. But for an even grid we must either take [-1.5, -.5, .5, 1.5] or [-2, -1, 0, 1]. Currently, the functions called "momentum_vector" and "position_vector" in _jellium.py return the second kind of grid. Because this grid has more negative momenta modes than positive ones, the imaginary components of the exponential in the dual basis external operator do not cancel in general, and we get complex diagonals, which means it is non-Hermitian. A similar problem happens in the plane wave basis. One might think a simple fix would be to change the grids to look like [-1.5, -.5, .5, 1.5] instead of [-2, -1, 0, 1].

However, I believe this issue came up once in the past and there was some fundamental reason why using the [-1.5, -.5, .5, 1.5] type of grid was problematic. @Spaceenter do you remember this? Your help / thoughts here would be appreciated.

[Spaceenter]

@babbush
Hi Ryan,
I searched for our old emails and I copy below your explanations to me at that time:
"You might think that if we have four modes, for instance, they would be [-1.5, -.5, .5, 1.5]. But I do not think that would work. Because the basis functions associated with those frequencies would have the wrong boundary conditions. This is because the frequency is really 2 pi * \nu / \Omega. So if \nu is not an integer, the plane wave doesn't have the same value at left and right sides of the box, which is very bad. Thus, instead, if we have four modes they should be [-2, -1, 0, 1]. Does this make sense?"

[babbush]

Yes, I did say that. However, I now realize that when we have a grid with an even number of points on it, the problem identified by @max-radin occurs and so [-2, -1, 0, 1] solution isn't acceptable for cases when one is considering an external potential with nuclei away from the lattice sites. Other than "feeling icky" I'm not completely certain that using the [-1.5, -.5, .5, 1.5] modes would be so bad (especially for the dual basis). @gkc1000 do you have any thoughts about this?

[babbush]

I spoke to Garnet, and he thinks there is not really a problem using momenta modes like [-1.5, -.5, .5, 1.5]. So we should try to see if this would work. Does anyone want to help make this change? @max-radin @Spaceenter @kevinsung any interest?

@idk3 any idea whether this will mess up the FFFT?

[babbush]

That's the main conceptual thing that needs to change but I suspect that other parts of the code rely on this in a way that will necessitate other changes. For instance, you will almost certainly also need to change the position_vector function in the analogous way and there are certain to be some tests that need to be adjusted after doing this.

[max-radin]

I tried shifting the grid (888af80), but this didn't result in a Hermitian operator. For example, in the plane-wave basis, terms like 0^ 0 end up having complex coefficients because k_{p-q} is non-zero when p = q. Any thoughts?

[babbush]

Hmm, I think k_{p-q} should be replaced then with (k_p - k_q). Then it must be zero when p = q. Is the complex coefficient on this number operator coming from the external potential? In general it must be possible to express the plane wave electronic structure Hamiltonian on even grids. We should perhaps try to figure out how to make this work by inspecting the equations. I have trouble seeing form the equations where the problem arises... do you? You mentioned you thought dualling (the method of aliasing) might be the source of the problem. Why did you think that?

[conta877]

has this been resolved? if not, I'm playing around with these - can look into it

[max-radin]

@babbush sorry I just realized that I never responded to your earlier questions! Yes, the complex number operators are coming from the external potential.

The reason why I thought the problem was due to dualling was that disabling the dualling yielded a Hermitian external potential operator in the plane-wave basis, even with a [-2, -1, 0, 1] type grid. (By disabling dualling, I mean replacing k_{q-p} with k_q - k_p; see 7fb605f.) I think the problem with dualling occurs when q - p equals half the maximum k-vector, i.e. k_q - k_p = k_max/2. In this case k_{q-p} = mod(k_q - k_p + k_max/2) - k_max/2 = -k_max/2 = k_{p-q}. But in order to get a Hermitian operator, you would need to have k_{q-p} = -k_{p-q}.

@conta877 I had started working on this a while ago but will be quite busy for the next few weeks, so if you're able to implement a solution that would be great!

[jarrodmcc]

Did anyone dive deeper into this? I will probably get to this as a side effect of something I plan to do this week for generalized unit cells and their reciprocal lattices. Two guesses I would make at a glance before working on this

1. The shifting of the grid as it's being called seems odd, because it's more correct to think of the k's as integer translations of the reciprocal lattice that satisfy a nice property of orthogonality with respect to integration over the supercell. If this is violated with half translations the basis isn't orthogonal any more, which could cause all kinds of problems.

2. A non-hermitian operator resulting from the aliasing procedure seems likely if the mod operation isn't consistently implemented for kp - kq and kq - kp. It needs to be done in a way that the resulting operator is always the same for both, which I'm not sure happens if you just mod the result for an uneven grid (I haven't looked at the current code, just a guess at how it's being done).

[conta877]

I looked at it a little this week. shifting is not a fix:; kp-kq are integers even if you shift the k's by 0.5 -> that destroys the fourier transforms.

Things to test: is it hermitian, are the FTs giving one another (pw, dual). Is the spectrum real.

here is an example problem: if you stick to integers for grid length = 2 you have (0,1). k0-k1 = -1 = 1 and k1-k0 = 1 which causes the hamiltonian to be non hermitian. Lets say we fix this issue by introducing a half shift, now the FT is not diagonal.

[jarrodmcc]

I think half integer shifts are a bad idea for many reasons including those I listed above. We shouldn't confuse the continuous variable that can be used for k-point sampling with functions that we really want to both have the correct periodicity and orthogonality relations. Everything so far is assuming Gamma point only as far as I can tell. Regarding the modulo choice, I'll look more into how this is done in typical electronic structure codes when this constraint is used. Without thinking too much about this, It feels like the condition should either only apply to the magnitude of the k vector, or include a factor of choice that comes from both the + and - parts of the exponential decomposition of cos so that the hermiticity always comes out as expected. What happens to the FFFT condition if we applied our version of the mod condition to only the upper half of the Hamiltonian p <= q, then filled in p<q with their complex conjugates it as it ~should~ turn out with the appropriate sign changes on the i's so that it's Hermitian?

…

On Sun, Mar 25, 2018 at 4:30 PM, Isil Ozfidan ***@***.***> wrote: I looked at it a little this week. its a bit of a mess; kp-kq are integers even if you shift the k's by 0.5 -> that destroys the fourier transforms. Things to test: is it hermitian, are the FTs giving one another (pw, dual). Is the spectrum real. here is an example problem: if you stick to integers for grid length = 2 you have (0,1). k0-k1 = -1 = 1 and k1-k0 = 1 which causes the hamiltonian to be non hermitian. Lets say we fix this issue by introducing a half shift, now the dual basis is not diagonal. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHXFPO2R04OfILaS2rB1tURbFmxTQ3pxks5tiCiLgaJpZM4SJ2s6> .

[conta877]

I agree with no half shifts. I tried the p>=p the FT is still not diagonal.

[jarrodmcc]

That's good information, thanks! I don't have a proof, but I guess it's possible this specific trick only really works for odd numbers of grid points. The diagonal FFFT requirement goes a bit beyond what a typical electronic structure code would need, in that the plane wave dual basis is a bit non-standard. This could be related to the conflict between ternary vs dyadic wavelets and symmetry requirements, which was an important point in Steve White's recent work on wavelets to construct diagonal hamiltonians, though I'm just vaguely conjecturing about that.

…

On Sun, Mar 25, 2018 at 6:52 PM, Isil Ozfidan ***@***.***> wrote: I agree with no half shifts. I tried the p>=p the FT is still not diagonal. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHXFPH0ONJCnzNuOq1XLCCFxnTBFPY7pks5tiEn5gaJpZM4SJ2s6> .

[conta877]

I have a feeling you are right. I tried a bunch of tricks and was gonna dive into it pen&paper next.

[babbush]

I could very well be wrong, but I am not entirely convinced that it is impossible to make all of this work with odd number grids. For what its worth, @gkc1000 has insisted to me on several occasions that everything to should extend to odd grids without a problem. I also find it very strange that the one-body operator is the only problematic part, and only when the nuclei are not centered on a grid point. Because for jellium everything works even on odd grids, right?

[jarrodmcc]

I think you mean work for even number grids, it seems to work for odd. It could be the case, but if we want to make it work, we're going to need more than proof by authority here I think, the definition of dualing we cite in the paper doesn't match the definition of dualing in the papers we cite(where it is defined by the grid used on the density relative to grid of the wavefunction). That in combination with using the PWD basis makes this case a bit odd, so I'm having a hard time finding precedent for our use case.

…

On Sun, Mar 25, 2018 at 7:26 PM, Ryan Babbush ***@***.***> wrote: I could very well be wrong, but I am not entirely convinced that it is impossible to make all of this work with odd number grids. For what its worth, @gkc1000 <https://github.com/gkc1000> has insisted to me on several occasions that everything to should extend to odd grids without a problem. I also find it very strange that the one-body operator is the only problematic part, and only when the nuclei are not centered on a grid point. Because for jellium everything works even on odd grids, right? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#213 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHXFPDrwLTdOdu4bsl7RSblqbENk4MmDks5tiFHZgaJpZM4SJ2s6> .

[babbush]

Right, I actually had forgotten which one wasn't working. But now that I think of it, I recall seeing that there is a way to make it work for even grids but I was never able to figure out how to do the Fourier transform in a way that made a unitary equivalence between plane wave and dual forms. I'll try to help think about this next week as well - it is rather important for where we're headed.

[gkc1000]

Since my mailbox is filling up ...

I'm not completely sure what is implemented inside OpenFermion, but:

1. Hermiticity is obviously a property of the operator i.e. we can choose an arbitrary set of basis functions and <m|H|n> = <n|H|m>*. So any possibility of observing non-Hermiticity can only arise from an approximation in computing the matrix element <m|H|n> (in relation to the earlier questions with max-radin).

2. The approximation that corresponds to the discrete-variable representation + dualing in the PRX paper can be interpreted for the nuclear potential V as

\sum_{i} <m|R_i><R_i|V|R_i><R_i|n>

where <m|R_i> is taken to be evaluation of the function at grid-point R_i, and the grid points are evenly spaced in the box. This is Hermitian.

3. if I look inside plane_wave_hamiltonian.py, it looks like the above is implemented in plane-wave space, i.e. one is working with the elements V(G_i) such that

V(R_i) = DFT[ V(G_i) ]

To ensure Hermiticity, one has to ensure that V(R_i) is real. This implies that V(G_i) = V(-G_i)* where G_i is "aliased" as in a discrete FT.

[jarrodmcc]

Thanks for the clarifications Garnet. I’m confident that the implementation can be made hermitian as you say; I guess the question is whether doing so with our other approximations immediately implies the diagonal property in the dual basis for any choice of integer grid. I’m only going off what people have written here(as opposed to the code or my own derivations) so far as I was planning to look into this this week as part of something else and wanted to get a feel for the current status, but your points will definitely be helpful in that investigation.

…

On Mar 25, 2018, at 7:58 PM, gkc1000 ***@***.***> wrote: Since my mailbox is filling up ... I'm not completely sure what is implemented inside OpenFermion, but: Hermiticity is obviously a property of the operator i.e. so we can choose an arbitrary set of basis functions and <m|H|n> = <n|H|m>*. So any possibility of observing non-Hermiticity can only arise from an approximation in computing the matrix element <m|H|n> (in relation to the earlier questions with max-radin). The approximation that corresponds to the discrete-variable representation + dualing in the PRX paper can be interpreted for the nuclear potential V as \sum_{i} <m|R_i><R_i|V|R_i><R_i|n> where <m|R_i> is taken to be evaluation of the function at grid-point R_i, and the grid points are evenly spaced in the box. This is Hermitian. if I look inside the plane_wave_hamiltonian.py, it looks like the above is implemented in plane-wave space, i.e. one is working with the elements V(G_i) such that V(R_i) = DFT[ V(G_i) ] To ensure Hermiticity, one has to ensure that V(R_i). This implies that V(G_i) = V(-G_i)* where G_i is "aliased" as in a discrete FT. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jarrodmcc]

This was fixed in #313 . The most logical way turned out to be defining the external potential in a symmetrized form in the dual basis that mimics what happens on an odd grid, and defining the external potential in the plane wave form through the FT of this definition. This lets us keep all the properties we most care about (diagonal form while being hermitian on both even and odd grids) while introducing minimal 'aliasing' as we've been calling it.