[Babase] Re: Agnonisim and grooming errors

Susan Alberts alberts at duke.edu
Thu Mar 15 11:55:39 EDT 2007


OK thanks for this input from both Leah and Karl. All very helpful in 
getting me to underestand the issues more. See below.

>On 03/15/2007 09:02:24 AM, Leah Gerber wrote:
>>Susan,
>>
>>I think Karl is mostly concerned about how to document this for people
>>who
>>will be using grooming and agonism data. Is it safe to say that prior
>>to
>>2007 we will have to assume that grooms are NOT on the true dates?


Yes probably. See more below.

>>For
>>
>>agonisms I think we can have an earlier data from which the dates are
>>correct.


Yes, probably, question is what date. See below.

>
>My concern is that the data be usable.  If, for whatever data/
>whatever time period, sometimes the date was forced to the
>first of the month and sometimes it wasn't then the person
>looking at that data _should_ ignore the day information.
>To get a reasonable answer they should force all the dates
>to the first of the month and analyze things that way.
>In which case we may as well force the data to the first
>of the month ourselves because the user is surely going
>to forget to do it and struggle (somewhat) when they
>must do it......
>
>If I knew when dates were forced to the first of the
>month and when not I could adjust the rules, and
>document....


I suspect, just based on what the data looked like, that even though 
the rule was to just enter first of the month for all data, the rule 
was not always followed. I don't know this for sure but it seems an 
obvious conclusion from what the data look like. So, I think we have 
three choices.

One is to do some digging and figure out exactly when the dates were 
entered as 1st and when not, and karl writes the rules accordingly. 
This seems like poor return on our time investment, but I haven't 
given it much thought.

Two is to force the dates to the 1st prior to some reasonable date. 
The way to decide the date would be to look at the actual data and 
figure out when the preponderance of dates were entered as 1st and 
when this stopped. This might not be very exact -- we would need to 
look at the data to see how clear this is. The test would be, when 
are the data reasonably evenly distributed across days of the month?. 
(Just for the record, there should not, in general, be fewer ad libs 
on the last days of the month prior to 1994; all days of the month 
were the same back then. After 1994, there might be fewer ag and 
groom on the very last day of the month (30th or 31st), which is 
always an office/meeting day -- but not fewer on the days before 
this.)

Three is to leave the data as is and make the user do the work. The 
accompanying documentatiuon would say something like "don't use dates 
for ag and groom data prior to xxxx". This might require the same 
level of work as option 2.

On the whole I like option 2 with the caveat that we probably cannot 
get a very good exact answer to this and we should not try. We should 
use the data to guide us to a reasonable date and go with that.

Thoughts?

Susan

>
>(Leah, FYI when looking at the conversion
>errors.  The rules as of the last conversion,
>v10, make no exceptions for psion data.  I've fixed this.)
>
>>
>>Susan Alberts <alberts at duke.edu>
>>Sent by: babase-bounces at eeblistserv.princeton.edu
>>03/15/2007 09:59 AM
>>Please respond to
>>The Baboon Database Project <babase at eeblistserv.princeton.edu>
>>
>>
>>To
>>The Baboon Database Project <babase at eeblistserv.princeton.edu>
>>cc
>>
>>Subject
>>[Babase] Re: Agnonisim and grooming errors
>>
>>
>>
>>
>>
>>
>>I think I am a little lost in this discussion regarding what we need
>>to accomplish. Please correct me on the below points or clarify as
>>needed. Sorry that I didn't retain the earlier text but I kept
>>getting confused about where to snip and what to keep.
>>
>>Karl provided the tables showing the distribution of ags and
>>groomings before 1995. The distribution is very odd, with most
>>entries occurring on 1st of the month. Question is, why is this?
>>
>>Part of the answer,  we definitely know, is that during some periods
>>of time, all agonism and grooming data were entered only on first of
>>month, regardless of when they actually were recorded.
>>
>>What we DON'T know is when this rule was and wasn't followed for both
>>types of data. So, we can't check to see whether the odd distribution
>>of days for G and A data correspond to our actual rules for when we
>>used first of month only versus exact date.
>>
>>What Karl's concern is (I think) is that the data are accurately
>>transferred from foxpro to postgresql. The weird distribution of
>>dates makes him concerned that there is a data transfer error that is
>>being obscured by our weird data entry patterns. karls, is this
>>correct?
>>
>>I doubt that we can accurately reconstruct from our memory the years
>>during which different types of data entry (with respect to which
>>dates were used in A and G records) were done, unless the records at
>>Princeton on this are very good. Tabby and Jeanne, what do you think?
>>
>>One brute force way to resolve the question is to do the query that
>>Karl, in Foxpro, did but break it down year by year and see if we can
>>identify years where different rules were followed. A even more brute
>>force way would be to just query on all "A" acts and order by date,
>>in Foxpro, and then do the same in postgresql. I just did this in
>>postgresql and found it instructive going through a couple years but
>>that will be very tedious. It does seem to be the case, from a VERY
>>quick glance, that whatever rules we implemented were not
>>consistently followed all the time.
>>
>>In order to decide how to proceed we need to decide how important it
>>is to reconstruct the rules of data entry over the years. It would
>>obviously be good to have this record. On the other hand, what is at
>>stake if we don't do this? I need to understand this.
>>
>>Susan
>>
>>
>>--
>>--------------------------------------------------------------------------------------------------------------------------------------
>>Susan Alberts, Department of Biology, Duke University, Box 90338,
>>Durham NC 27708
>>919-660-7272 (phone), 919-660-7293 (FAX)
>>_______________________________________________
>>Babase mailing list
>>Babase at www.eco.princeton.edu
>>http://www.eco.princeton.edu/mailman/listinfo/babase
>>
>
>>_______________________________________________
>>Babase mailing list
>>Babase at www.eco.princeton.edu
>>http://www.eco.princeton.edu/mailman/listinfo/babase
>>
>
>
>Karl <kop at meme.com>
>Free Software:  "You don't pay back, you pay forward."
>                  -- Robert A. Heinlein
>
>_______________________________________________
>Babase mailing list
>Babase at www.eco.princeton.edu
>http://www.eco.princeton.edu/mailman/listinfo/babase


-- 
--------------------------------------------------------------------------------------------------------------------------------------
Susan Alberts, Department of Biology, Duke University, Box 90338, 
Durham NC 27708
919-660-7272 (phone), 919-660-7293 (FAX)


More information about the Babase mailing list