[Babase] Update trouble - Demog notes cannot have no note (no
nulls allowed)?
Niki Learn
nlearn at princeton.edu
Tue Sep 7 13:46:46 EDT 2010
-----Original Message-----
From: babase-bounces at eeblistserv.Princeton.EDU
[mailto:babase-bounces at eeblistserv.Princeton.EDU] On Behalf Of Karl O. Pinc
Sent: Tuesday, September 07, 2010 1:02 PM
To: The Baboon Database Project
Subject: Re: [Babase] Update trouble - Demog notes cannot have no note (no
nulls allowed)?
On 09/07/2010 11:22:14 AM, Niki Learn wrote:
> Yes, most of the M records in babase seem to be unnecessary and there
> are at
> least a small number that conflict with demography notes that are not
> yet in
> babase. I expect I will delete most of these Ms when I do the demog
> note
> backfill entries. There do, however, seem to be a small number of
> "real"
> Manual entries where, for example, we don't have an actual census
> datum but
> we know that an animal was in another group prior to arriving in a
> study
> group and so we manually place the animal in that other group prior
> to
> their
> arrival in a censused group (such as when a previously unknown animal
> is
> seen coming from another group or an easily recognizable but unnamed
> animal
> known to be in a peripheral group shows up in a study group). The M
> then
> provides a sense that the datum is a result of interpretation of
> circumstantial evidence rather than being an actual census point.
You probably want an 'N' in these cases, because 'N' does not
interpolate whereas 'M' does.
Um...? I've never used N. There are only two baboons in babase with N
records, all from the early '90s. I have no idea why they have those (each
of the two has a bunch of them). Maybe when I am not in the midst of update
I will look them up and see if there is any record of a reason.
I don't see a problem with having interpolated points for the scenarios I
mentioned above. In the case of an animal being seen joining from a
nonstudy group, they would get the M record for the day before - since this
scenario typically happens in the morning when the groups have slept near
each other, this is pretty straightforward and I don't think interpolating
that manual entry is any different from interpolating any other census
point. Ditto for an animal that is known to have been in a nonstudy group
before arriving. Occasionally (as recently happened with a couple animals
that had been in Olkenya group), the team has actually included a descriptor
of the known animal in the other group census data so you can go back and
apply their new name to their record retroactively, but at other times we
just know that they were in the group and don't have any specific census
records for them in that group (such as with the same males when they were
in Stud's prior to joining Olkenya). So what I have been thinking works
best is to give them a manual census record on the last day they were absent
from the study group, since that is the last date on which we know they were
not there - again interpolation between last absence and first presence in
the study group seems perfectly reasonable.
I'm really not sure when you would want to use N. Practically speaking, it
is my impression that M has been used like N in that "it is presumed to be
the result of some analysis" even though the documentation only says that
about N. Btw, to make a manual entry I enter it in one of the normal
fashions (census or demog) and then change it to manual, so I'm thinking
this part of the definition - "when there is no other way to get the data
in" - must be for babase 1.0 and no longer relevant?
I'm more puzzled/concerned by how to get babase to interpolate correctly
when animals are seen in two groups on the same day. I have to pick one
group for them to be in (and there doesn't seem to be any rule about which
to pick) but then they get interpolated on either side of that even though
we know they were in the other group the day before (at least in some cases,
like if they slept with that group and then came over to the other group).
This is another place we should be using M but where I think it has only
been used sporadically. Perhaps the rule would be to put them in the second
group they were seen in but give them an M record in the first group the day
before? Also, I find it annoying that nonstudy group animals keep reverting
to group 9 in between censuses when the censuses are too far apart (as they
often are). I guess they could be moving around during the time we're not
seeing them, but they could be doing that during the time they do get
interpolated too... So it seems like it would be nice for them to stay in
the nonstudy group between censuses if they are present at both ends. But I
guess it is too complicated to have different rules for animals in different
groups (with some groups changing category over time).
> Thanks for allowing comment to be null!
I exist to make your life complicated. Oh no wait, I mean
to serve. :)
I guess we all have a role in life... ;)
> Niki
>
> -----Original Message-----
> From: babase-bounces at eeblistserv.Princeton.EDU
> [mailto:babase-bounces at eeblistserv.Princeton.EDU] On Behalf Of Karl
> O.
> Pinc
> Sent: Tuesday, September 07, 2010 11:35 AM
> To: The Baboon Database Project
> Subject: Re: [Babase] Update trouble - Demog notes cannot have no
> note
> (no
> nulls allowed)?
>
> Hi,
>
> It's taken a while to get through my email backlog.
>
> It has also taken me a while to work through the implications
> of having a NULL DEMOG.Comment. My analysis is below and
> is probably worth reading.
>
> In the end I came to the conclusion that everyone has made
> the right decision and a NULL DEMOG.Comment is the right
> way to go. The change has been made to the database
> and the documentation will be updated very shortly.
>
> ----------------<snip>-------------
>
> Allowing the demography note to be null is not the right way
> to go. Instead of doing that we should remove the DEMOG
> table entirely. But I don't think that's the right approach
> either.
>
> I'm not sure exactly what the problem is that you're having,
> or more precisely what exactly it is that you want to accomplish,
> so let me list the various issues that I see. I'm going to think
> out loud as I go.
>
> There are two possible sides to the problem. A data entry problem
> and a record keeping problem. Record keeping is paramount,
> although of course the data must go in, so let me start with
> record keeping. Once we know what we want to do with that
> we can have further discussion regarding data entry.
>
> There are a couple of possibilities. First, you have no regular
> census
> but do have a demography note, with a useless remark. This is
> exactly
> the sort of case where you would use the 'M' (manual) CENSUS.Status
> value -- creating a manual census entry. The 'M' Status was used
> prior to 2001, I think by Daphine in an attempt to force
> interpolation
> to do the right thing when we discovered that our early stabs
> at interpolation were not working. However the Status has not
> been used since so, if it's really necessary to record that the
> entry is the result of a demography note it could be used to mean
> 'census entry created as a result of a demography note'. (In any
> case I've a feeling that all those pre-2001 "M" statuses don't belong
> and should be removed. I could be wrong in this.)
>
> This brings us to the heart of the issue, whether it's necessary
> to record that the source of the CENSUS row is a demography note.
> If it's not necessary to note that the source
> of the CENSUS-row-that-does-not-appear-on-the-census-sheet is
> a demography note then 'M' works perfectly well. Alternately,
> again, if there's no other reason for manually fussing with CENSUS
> then the 'M' rows must be there because of demography notes.
> If there are multiple sources of census-like data, demography notes,
> something else, etc., that would cause you to make changes to CENSUS
> then we could either create separate codes for each, which we would
> want to do if it was important to query on this information, or
> we could have a comment column that indicates the information's
> source. In the latter case, that's pretty much exactly what
> the DEMOG.Comment column is -- and you'd enter in that column
> something like "A demography note records that so-and-so
> is in such-and-such group". Exactly the comment which
> started this thread, the comment that you want to get rid of!
>
> :-)
>
> (FWIW, I've a dim recollection of having had a discussion
> like this one before.)
>
> The second possibility is that there is already a CENSUS row
> for the individual and there is a demography note placing
> the individual in the group. In this case I think you do
> want to keep track of the existence of the demography note
> so that if the record of the census gets changed or otherwise
> goes away you need to keep a row on CENSUS because the
> demography note exists. Humm.... The right way to do
> this is to allow NULL demography notes.
>
> On 09/03/2010 04:35:55 PM, Niki Learn wrote:
> > Yes, mostly it is with other group censuses. There may also be
> > occasional
> > instances where a baboon is marked present using a demography note
> > during an
> > incomplete census.
> >
> > Thanks - Karl, please fix at your earliest convenience.
> >
> > Niki
> >
> > -----Original Message-----
> > From: babase-bounces at eeblistserv.Princeton.EDU
> > [mailto:babase-bounces at eeblistserv.Princeton.EDU] On Behalf Of
> Susan
> > Alberts
> > Sent: Friday, September 03, 2010 5:31 PM
> > To: The Baboon Database Project
> > Subject: Re: [Babase] Update trouble - Demog notes cannot have no
> > note
> > (no
> > nulls allowed)?
> >
> > I don't completely understand what situations produce a demog note
> > that has a group and an ID number but no explanation-I presume that
>
> > this arises with other groups censuses? And sometimes with regular
> > group censuses?
> >
> > If I'm understanding correctly then I agree with Jeanne; go with 1.
> >
> > Susan
> > On Sep 3, 2010, at 5:06 PM, Niki Learn wrote:
> >
> > > Jeanne thinks 1 is best. (2 is clunky and 3 is bad)
> > >
> > > If that's okay by Susan as well, Karl please remove the not-null
> > > constraint on the comment column in demog. Then the upload can
> > > commence.
> > >
> > > Thanks!
> > > Niki
> > >
> > > From: babase-bounces at eeblistserv.Princeton.EDU
> > [mailto:babase-bounces at eeblistserv.Princeton.EDU
> > > ] On Behalf Of Niki Learn
> > > Sent: Friday, September 03, 2010 4:26 PM
> > > To: 'The Baboon Database Project'
> > > Subject: [Babase] Update trouble - Demog notes cannot have no
> note
>
> > > (no nulls allowed)?
> > >
> > > I am working on the upload in test and have just come to the
> > > demography notes portion. Demography notes also include census
> > > entries and other groups notes entries on nonstudy groups, where
> > > often the "note" would just consist of "so-and-so is present in
> > such-
> > > and-such group".
> > >
> > > At the joint lab meeting we talked extensively about some
> revisions
> >
> > > to how we enter demography notes and what actually counts as a
> > > demography note. One of the things we discussed was that these
> > > notes should be as un-wordy as possible and that the ones that
> just
> >
> > > say things like "so-and-so is present in such-and-such group",
> > which
> >
> > > is already captured in the census record the demography note
> > > creates, that we didn't have to have any words because they were
> > > redundant. So I implemented that for this upload and, just on
> the
>
> > > first group, got this error 9 times:
> > >
> > > CAUTION -- This error may not be real; prior row(s) of the
> uploaded
> >
> > > file were rejected:
> > > Line 17: ERROR: null value in column "comment" violates not-null
> > > constraint
> > >
> > > So, the note can't be blank. I see three possible options
> > >
> > > 1) Remove the rule saying that the comment column (the
> actual
>
> > > note) cannot be null. (This seems simplest.)
> > > 2) Go back to "so-and-so is present in such-and-such group".
>
> > > (Perhaps preferred for some reason?)
> > > 3) Or I could enter those other groups entries the way we
> > enter
> >
> > > census data - it just might have a lot of "N" entries in the file
> I
> >
> > > upload (but babase reads "N" as "don't create a record for this
> > > animal on this date" so it wouldn't create extra records or
> > anything
> >
> > > like that). The downside there being that if there were actual
> > > demography notes to go with any of them I would need a census
> > record
> >
> > > and a demography note but it comes out the same in babase. Also,
>
> > > entering them as demog notes tells you the reference source
> (i.e.,
>
> > > is there an actual census sheet in the other groups binder or is
> it
> >
> > > an other groups note within one of the study group binders and
> > which
> >
> > > study group is it with).
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Niki
> > >
> > >
> > > _______________________________________________
> > > Babase mailing list
> > > Babase at www.eco.princeton.edu
> > > http://www.eco.princeton.edu/mailman/listinfo/babase
> >
> > ------------------------------------------------------------
> > Susan Alberts, Professor of Biology, Duke University, Box 90338,
> > Durham NC 27708. Phone 919-660-7272, FAX 919-660-7293
> >
> >
> >
> >
> >
> > _______________________________________________
> > Babase mailing list
> > Babase at www.eco.princeton.edu
> > http://www.eco.princeton.edu/mailman/listinfo/babase
> >
> > _______________________________________________
> > Babase mailing list
> > Babase at www.eco.princeton.edu
> > http://www.eco.princeton.edu/mailman/listinfo/babase
> >
> >
>
>
>
>
> Karl <kop at meme.com>
> Free Software: "You don't pay back, you pay forward."
> -- Robert A. Heinlein
>
>
> _______________________________________________
> Babase mailing list
> Babase at www.eco.princeton.edu
> http://www.eco.princeton.edu/mailman/listinfo/babase
>
> _______________________________________________
> Babase mailing list
> Babase at www.eco.princeton.edu
> http://www.eco.princeton.edu/mailman/listinfo/babase
>
>
Karl <kop at meme.com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
_______________________________________________
Babase mailing list
Babase at www.eco.princeton.edu
http://www.eco.princeton.edu/mailman/listinfo/babase
More information about the Babase
mailing list