[Babase] Auto updating of PREGS.Resume, new conversion files, new errors

Jeanne Altmann altj at Princeton.EDU
Fri Sep 22 10:38:05 EDT 2006


And one more YAY! 

-----Original Message-----
From: babase-bounces at eeblistserv.princeton.edu
[mailto:babase-bounces at eeblistserv.princeton.edu] On Behalf Of Karl O.
Pinc
Sent: Thursday, September 21, 2006 7:30 AM
To: babase at eeblistserv.princeton.edu
Subject: [Babase] Auto updating of PREGS.Resume, new conversion
files,new errors

Hi,

It appears I have all the validation working, and all the automatic
updating working but for cycstats and repstats.  (As previously
discussed.)

Yay!  It turns out there was quite a bit of trigger code throughout the
sexual cycle tables that had not been exercized and had bugs.

In getting all the validation and automatic updating working it proved
easiest to have PREGS.Resume automatcially become the resuming Tdate.
See the PREGS docs for more info.

There's a new set of conversion errors with all the validation turned
on, but only loading the sexual cycle, interaction, and point sample
data as has been done lately.  The new files are:

papio.biology.duke.edu:/biology/groups/babase/errors/loadscript.pregs
papio.biology.duke.edu:/biology/groups/babase/errors/errors.pregs

The latter file does not yet exist.  It will come into existance in an
hour or so when the conversion finishes.

(Leah and Tabby, please delete whatever files you are not using in
papio.biology.duke.edu:/biology/groups/babase/errors/.
Some of the files in there are large and while I don't know that there's
a problem, I don't want to run out of space at an awkward
moment.)

Tabby can look at the sexual cycle errors and see what they are about.
As usual, there's a lot of "cascading" going on.  See below for some
analysis.

As Tabby and I discussed, there are 3 new files that supply input to the
conversion.  They contain SQL statements (mostly INSERTS).

The first is:
papio.biology.duke.edu:/biology/groups/babase/real_gaps

This contains the contents of the CYCGAPS table, as we would like it to
look after the conversion.  This (and the other files) was (were)
initially generated programatically, but there's since been manual
changes and we may want to make further changes.  (For instance, one
change was made as a result of one of Catherine's findings where there
was a CYCPOINTS data point one day before observation resumed, so the
data was changed so that the observation resumed one day earlier for
that individual.)

The second and third are:
papio.biology.duke.edu:/biology/groups/babase/fake_gaps
papio.biology.duke.edu:/biology/groups/babase/undo_fake_gaps

The Babase rules won't allow a pregnancy to have a resume cycle unless
there's a BIOGRAPH.Pid value that references the PREGS row.  And they
won't allow cycling after a pregnancy without a resume unless there's a
gap.
And of course a PREGS must have a Conceive on CYCPOINTS.  So, in order
to load PREGS the conversion program first loads CYCPOINTS.
This allows PREGS to have a Conceive.
Then it temporarly creates a ("fake") gap on each birthday, allowing
pregs with no Resume's to have subsqeuent CYCPOINTS. Then it loads
PREGS, but without any Resume values because there are no births yet.
Then it puts the PREGS pids in BIOGRAPH.Pid to get a birth.  Then it
puts the PREGS.Resumes in.  And finally it removes all the temporary
gaps it created just so it can get the PREGS in.

The fake_gaps file contains SQL INSERT statements that create all the
"temporary" gaps.  The undo_fake_gaps file contains the SQL to remove
the "temporary" gaps.

(The SQL in the undo_fake_gaps file also has statements that check to
see that the gap to be removed exists, and if not it produces a warning.
This is a double-check to make sure we don't think we're removing a
temporary gap and then fail to remove it.  Leah might be interested in
the SQL techniques in the 2 "fake" files.)

So, if pregancies are moved about then the fake_gaps and undo_fake_gaps
files might need adjusting.

There is already an issue with about 20 of the fake gaps.  They are for
stillbirths.  The date of the stillbirth is the female's next Tdate.
Babase does not care if you want to do this, I don't think (somebody
might want to check the docs), but the conversion program does care
because the fake gap is on the "birthdate" and that means there's a
CYCPOINTS row (the Tdate) that occurs during a gap in observation.  This
is a no-no.  The conversion produces an error that says "Cannot have
CYCPOINTS when there's no observation".  The solution is probably to
adjust the date of the fake gap to be (say a week) before the
(still)birthdate where there's a problem.  I leave that to Tabby if
that's really what should be done.

Here's the mothers that I took the time to look at in detail that have
this problem with stillbirths.  (There are some others with the above
error message but I didn't take the time to check them.
Tabby should check them an confirm that
what I think is going on is really happening.):

NZI 1993-03-31
SIS 1988-12-18
LUN 1995-10-18
OCH 1997-03-29
VIN 1997-10-29
VIN 1997-12-14
VIN 1998-01-16

Another issue with the "fake" gaps is that the State of the gap must not
conflict with the neighboring CYCPOINTS rows.
I've code in the fake_gaps file to compute the common case but there's
also some cases where I've edited the fake_gaps file and altered a
particular data value.  Changes to CYCPOINTS might require changing the
State of the fake gap.

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                  -- Robert A. Heinlein

_______________________________________________
Babase mailing list
Babase at www.eco.princeton.edu
http://www.eco.princeton.edu/mailman/listinfo/babase



More information about the Babase mailing list