Have `treeSeqRememberIndividuals` actually remember individuals? about slim

[petrelharp]

I guess I'm proposing:

Individuals would only be part of the individual table forevermore (i.e., permanently remembered) if treeSeqRememberIndividuals() had been called on them. Usually, these individuals would have information as it was at the time that treeSeqRememberIndividuals() was called. However, I am proposing that if a permanently remembered individual is alive when the tree sequence is written out to file, then their information will be whatever it is at that time.

However, if a tree sequence is written out again in the future when that individual is not alive, then their information in the individual table will be as it was when treeSeqRememberIndividuals was called, because that's what's in the remembered_individual_table, and the output individual table starts from a copy of this. We'd at least be able to tell this had happened, because in the tree sequence produced by the first call, the individual is marked as alive.

For example:

1. Generation 10: remember individual 1, who is age 3.

   • in remembered_individual_table, individual 1 is age 3.

2. Generation 12: output tree sequence; individual 1 is still alive.

   • in remembered_individual_table, individual 1 is age 3.
   • in the output tables, individual 1 is age 5 and marked as alive

3. Generation 20: output tree sequence, individual 1 has died.

   • in remembered_individual_table, individual 1 is age 3.
   • in the output tables, individual 1 is age 3 and not marked as alive.

[petrelharp]

Oh! We did this!

[bhaller]

We'd need a policy for what happens if someone got remembered twice (keep their data from the first time, maybe?).

There is also a bit of an odd question when it comes to individuals that are both remembered and alive. Does their entry contain the information at the time point when they were remembered, even if it has subsequently changed? Presumably not – presumably entries for living individuals contain the current information for those individuals. So then once that individual dies, does its entry then revert to the information that was present at the time it was remembered? That's pretty weird. There is always this tension, with the tree-seq tables, with mutable information, unfortunately. Between "up to date info" versus "info at time of recording".

[petrelharp]

Does their entry contain the information at the time point when they were remembered, even if it has subsequently changed? Presumably not – presumably entries for living individuals contain the current information for those individuals.

Oh dear - yes, it has to, to make loading back in of the .trees file work. So, then I propose that the policy be that the most recent time they were remembered (which includes writing out the tree sequence file!) be what is recorded in the individual table.

[bhaller]

So, then I propose that the policy be that the most recent time they were remembered (which includes writing out the tree sequence file!) be what is recorded in the individual table.

Are you proposing that writing out a file would permanently remember all of the extant individuals, as if treeSeqRememberIndividuals() had been called on them? Or just that they would be considered "remembered" in terms of this most of "most recent time remembered", for what info gets written out?

[bhaller]

Yes, I think this is the way it needs to work. It's a bit strange, but seems best overall.

[bhaller]

OK, reviewing this. You wrote:

Generation 12: output tree sequence; individual 1 is still alive.
in remembered_individual_table, individual 1 is age 3.
in the output tables, individual 1 is age 5 and marked as alive

So individual 1 is in remembered_individual_table, recorded with an age of 3, and your proposal is that the output table would always begin with remembered_individual_table. So I guess you are proposing that the output table would contain individual 1 at its assigned position near the beginning, within the remembered_individual_table section, but that its age entry would get patched, at save time, with its current age at time of saving, and its alive flag would be patched to true (whereas entries in the remembered_individual_table would presumably all have false, so that we don't have to maintain those values as the simulation runs)? Does this sound right?

[petrelharp]

Yes, exactly.