[Babase] Auto updating of PREGS.Resume, new conversion files, new errors

Karl O. Pinc kop at meme.com
Thu Sep 21 07:30:17 EDT 2006


Hi,

It appears I have all the validation working, and all
the automatic updating working but for cycstats
and repstats.  (As previously discussed.)

Yay!  It turns out there was quite a bit of
trigger code throughout the sexual cycle tables that had not
been exercized and had bugs.

In getting all the validation and automatic updating
working it proved easiest to have PREGS.Resume
automatcially become the resuming Tdate.  See the
PREGS docs for more info.

There's a new set of conversion errors with all the
validation turned on, but only loading the sexual
cycle, interaction, and point sample data as
has been done lately.  The new files are:

papio.biology.duke.edu:/biology/groups/babase/errors/loadscript.pregs
papio.biology.duke.edu:/biology/groups/babase/errors/errors.pregs

The latter file does not yet exist.  It will come
into existance in an hour or so when the conversion
finishes.

(Leah and Tabby, please delete whatever files
you are not using in
papio.biology.duke.edu:/biology/groups/babase/errors/.
Some of the files in there are large and
while I don't know that there's a problem,
I don't want to run out of space at an awkward
moment.)

Tabby can look at the sexual cycle errors and see
what they are about.  As usual, there's a lot of
"cascading" going on.  See below for some analysis.

As Tabby and I discussed, there are 3 new files that
supply input to the conversion.  They contain
SQL statements (mostly INSERTS).

The first is:
papio.biology.duke.edu:/biology/groups/babase/real_gaps

This contains the contents of the CYCGAPS table, as we
would like it to look after the conversion.  This
(and the other files) was (were)
initially generated programatically, but there's
since been manual changes and we may want to make further
changes.  (For instance, one change was made as a result
of one of Catherine's findings where there was a CYCPOINTS
data point one day before observation resumed, so the
data was changed so that the observation resumed one day
earlier for that individual.)

The second and third are:
papio.biology.duke.edu:/biology/groups/babase/fake_gaps
papio.biology.duke.edu:/biology/groups/babase/undo_fake_gaps

The Babase rules won't allow a pregnancy to have
a resume cycle unless there's a BIOGRAPH.Pid
value that references the PREGS row.  And they
won't allow cycling after a pregnancy
without a resume unless there's a gap.
And of course a PREGS must have a Conceive
on CYCPOINTS.  So, in order to load PREGS
the conversion program first loads CYCPOINTS.
This allows PREGS to have a Conceive.
Then it temporarly creates a ("fake") gap on each birthday,
allowing pregs with no Resume's to have subsqeuent
CYCPOINTS. Then it loads PREGS, but without any Resume values
because there are no births yet.
Then it puts the PREGS pids in BIOGRAPH.Pid to
get a birth.  Then it puts the PREGS.Resumes
in.  And finally it removes all the temporary
gaps it created just so it can get the PREGS
in.

The fake_gaps file contains SQL INSERT statements
that create all the "temporary" gaps.  The
undo_fake_gaps file contains the SQL to remove
the "temporary" gaps.

(The SQL in the undo_fake_gaps file also has statements
that check to see that the gap to be removed exists,
and if not it produces a warning.  This is a double-check
to make sure we don't think we're removing a temporary gap
and then fail to remove it.  Leah might be interested
in the SQL techniques in the 2 "fake" files.)

So, if pregancies are moved about then the fake_gaps
and undo_fake_gaps files might need adjusting.

There is already an issue with about 20 of the
fake gaps.  They are for stillbirths.  The date
of the stillbirth is the female's next Tdate.
Babase does not care if you want to do this,
I don't think (somebody might want to check
the docs), but the conversion program does care
because the fake gap is on the "birthdate" and
that means there's a CYCPOINTS row (the Tdate)
that occurs during a gap in observation.  This
is a no-no.  The conversion produces an error
that says "Cannot have CYCPOINTS when there's
no observation".  The solution is probably to
adjust the date of the fake gap to be (say
a week) before the (still)birthdate where
there's a problem.  I leave that to Tabby
if that's really what should be done.

Here's the mothers that I took the time to
look at in detail that have this problem
with stillbirths.  (There are some others
with the above error message
but I didn't take the time to check them.
Tabby should check them an confirm that
what I think is going on is really happening.):

NZI 1993-03-31
SIS 1988-12-18
LUN 1995-10-18
OCH 1997-03-29
VIN 1997-10-29
VIN 1997-12-14
VIN 1998-01-16

Another issue with the "fake" gaps is that the State of the gap
must not conflict with the neighboring CYCPOINTS rows.
I've code in the fake_gaps file to compute the common case
but there's also some cases where I've edited the fake_gaps file
and altered a particular data value.  Changes
to CYCPOINTS might require changing the State of the fake gap.

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                  -- Robert A. Heinlein



More information about the Babase mailing list