Implimentation schedule -- Was: Re: [Babase] Merging JPSAMPS
with FPSAMPS (and JPNEIGHBORS with FPNEIGHBORS)
Susan Alberts
babase@www.eco.princeton.edu
Mon, 11 Oct 2004 06:49:06 +0300
This is very helpful and clear, thanks.
The idea that we will be switching over to the new system in Nov is
really great, I'm very excited about it.
It sounds as though you need Hunter to move on installing
phpgsqladmin - do you need any help making that happen? I could email
him. Is the front end necessary for the rest of week 1 to happen?
It also sounds as though Steph in particular will have a busy week,
and your progress will depend a lot on her being able to query the
tables in the way you request this week? Steph, please let us know
if you have any questions about your role in this -- perhaps you and
Karl have already talked and you know about these queries but perhaps
you have questions and will need to talk?
I don't know which programs are and are not currently used, Daphne
and Steph will provide input on that.
With respect to other things we need to do after your final step (add
new tables and columns and extend database), we also talked about
looking at the front end and thinking about what if anything we might
want to do to tailor it more to our needs. This is an issue that we
will probably want to revisit in a couple of weeks. We also talked
about making a web-accessed visitor site to the database that gives
visitors limited access to our data. We get some pressure to make our
data "public" and i have some ideas that i think/hope will be easy to
implement and will have some bells and whistles attached that will
meet this need without compromising our control of our data. I think
Jeanne probably has some ideas too, and others may as well. This is
an issue that we will want to talk about in the next month or two, to
get the shape of work for the next few months clear in our minds.
Susan
>
>
>In my suggested order:
>
>Week 1 (Oct 10)
>
>I need to speed up the conversion, really that means speed up the
>interpolation
>and speed up the repstats/cycstats while I'm at it. I expect to be
>done by Monday night, so we should have a converted database by Tue.
>
>Hunter needs to put in a front end (phppgsqladmin) so the database
>can be accessed from the web. (Your postdoc could use Unix, and
>might very well find it useful to script their queries, but
>a web interface would be a more gentle introduction.)
>
>We need to check the converted database against the foxpro database,
>line by line with automated tools. Steph has to query each table
>in Foxpro and transfer the result to:
>
>login.biology.duke.edu:/biology/groups/babase/foxdata/
>
>One query per table please. They must be sorted so each row
>winds up in a well defined place. I should look at the queries
>beforehand and will need them to duplicate in postgresql.
>(Mostly they'll be identical SQL, but there's a little bit of
>difference between the old and new table structure.)
>Might as well give the files the name of the table.
>
>I need to randomize the foxpro census data as it's fed to
>the conversion program and re-run that part of the conversion
>several (7) times comparing results to be sure the new interpolation
>routines work irrespective of input ordering.
>
>Steph & Daphine deal with any other data issues.
>
>* Conversion programming done. Data can be queried.
>
>Week 2 (Oct 17) & 3 (oct 24):
>
>The programs need rewriting:
>
>addcen.prg
>addmemb.prg
>appender.prg
>buildmembers.prg (done)
>convcen.prg
>convint.prg
>gwhere.prg
>incen.prg
>interp.prg (done)
>iwhere.prg
>julian.prg (done)
>makerep.prg (done)
>newincen.prg
>ranker.prg
>reinterp.prg (done)
>subrank.prg
>triprpt.prg
>tst.prg
>upcen.prg
>updemog.prg
>upinter.prg
>uppoints.prg
>valcen.prg (obsolete/done (in db))
>valdemog.prg (obsolete/done (in db))
>valinter.prg (obsolete/done (in db))
>AGE.PRG (obsolete/postgresql does it)
>COMPARE.PRG
>FEMFECUN.PRG
>INSROW.PRG
>INTEGRIT.PRG (Mostly done, needs review)
>MERGEIP.PRG
>minutes.PRG (obsolete/postgresql does it)
>RNKDATE.PRG (done)
>SMUSH.PRG
>UNJULIAN.PRG (done)
>UPCYCLE.PRG
>UPMEMB.PRG (done)
>UPSEQ.PRG
>UPSUPERG.PRG
>VALCYCLE.PRG (obsolete/done (in db))
>VALMEMB.PRG (obsolete/done (in db))
>ZEROS2.PRG
>ZEROS.PRG
>
>Offhand I can't remember what all of these do. If you're not using
>any let me know.
>
>Plan is to do the ranker first. That could take a week. The
>rest should be easy and take another week. This is where there
>begins to be something on the web site.
>
>http://biology.duke.edu/babase/
>
>Maybe we test by doing input work twice, in both the
>new and old systems and then re-running the comparison test?
>
>* Sometime after this we cut-over to the new system.
>
>Week 4 (Oct 31):
>
>The 'reminder' system that reports warning and errors and allows
>them to be either ignored forever or put off until later.
>The queries are done to generate the warnings/errors, probably,
>but I need to do the rest.
>
>Week 5+ (Nov 7):
>
>Karl goes to Princeton for 2 weeks.
>
>Document the new system. This is a big-ish job (week?(s?)) involving going
>over all the old docs and checking things. While I'm at it
>I'll be converting the format and putting it on the web/into pdf.
>
>Work with staff. Make 'standard' queries. Maybe make some views.
>Iron out problems.
>
>Get rid of the babase_sandbox database and switch over to database
>schemas -- put a sandbox area in all databases that students
>et-al can make tables in that are kept separate from the babase
>tables but where both sandbox and babase tables can be queried
>together.
>
>* Done with Babase 2.0!
>
>Convert to text all the old foxpro datasets and other stuff used for input,
>just to be able to keep them for the future.
>
>Final archive of old foxpro system, with a separate archive of programs
>for reference.
>
>* Done with Foxpro!
>
>Add new columns/tables and extend babase.
>
>March boldly into the future.
>
>The only thing I've left out here is the party, but maybe somebody
>else can think of something.
>
>Karl <kop@meme.com>
>Free Software: "You don't pay back, you pay forward."
> -- Robert A. Heinlein
>_______________________________________________
>Babase mailing list
>Babase@www.eco.princeton.edu
>http://www.eco.princeton.edu/mailman/listinfo/babase
--
--------------------------------------------------------------------------------------------------------------------------------------
Susan Alberts, Department of Biology, Duke University, Box 90338,
Durham NC 27708
919-660-7272 (phone), 919-660-7293 (FAX)