Implimentation schedule -- Was: Re: [Babase] Merging JPSAMPS with FPSAMPS (and JPNEIGHBORS with FPNEIGHBORS)

Karl O. Pinc babase@www.eco.princeton.edu
Sun, 10 Oct 2004 21:10:57 -0500


On 2004.10.09 09:12 Susan Alberts wrote:

> i certainly understand why you'd rather not come this month, I can 
> appreciate how hard it must be to change location all the time. As 
> you know my main concern is that we maintain momentum on the 
> database. My new demography postdoc is arriving  this week, and i had 
> always envisioned that the conversion would be complete and the new 
> postgresql database up and running before he arrived. This obviously 
> isn't the case and I guess I haven't yet sorted out the implications 
> of this in terms of his work. I guess my feeling is that we should 
> just keep tabs on progress.
> 
> Perhaps it would be helpful if you sort of summarized where we are 
> right now -- i for one have lost track a bit of exactly where we are 
> versus where we wanted to be. What do we have left to do before we 
> have a fully functional postgresql database?

I think we're on track, but don't recall exactly where we wanted to be.

In my suggested order:

Week 1 (Oct 10)

I need to speed up the conversion, really that means speed up the 
interpolation
and speed up the repstats/cycstats while I'm at it.   I expect to be
done by Monday night, so we should have a converted database by Tue.

Hunter needs to put in a front end (phppgsqladmin) so the database
can be accessed from the web.  (Your postdoc could use Unix, and
might very well find it useful to script their queries, but
a web interface would be a more gentle introduction.)

We need to check the converted database against the foxpro database,
line by line with automated tools.  Steph has to query each table
in Foxpro and transfer the result to:

login.biology.duke.edu:/biology/groups/babase/foxdata/

One query per table please.  They must be sorted so each row
winds up in a well defined place.  I should look at the queries
beforehand and will need them to duplicate in postgresql.
(Mostly they'll be identical SQL, but there's a little bit of
difference between the old and new table structure.)
Might as well give the files the name of the table.

I need to randomize the foxpro census data as it's fed to
the conversion program and re-run that part of the conversion
several (7) times comparing results to be sure the new interpolation
routines work irrespective of input ordering.

Steph & Daphine deal with any other data issues.

* Conversion programming done.  Data can be queried.

Week 2 (Oct 17) & 3 (oct 24):

The programs need rewriting:

addcen.prg
addmemb.prg
appender.prg
buildmembers.prg (done)
convcen.prg
convint.prg
gwhere.prg
incen.prg
interp.prg (done)
iwhere.prg
julian.prg (done)
makerep.prg (done)
newincen.prg
ranker.prg
reinterp.prg (done)
subrank.prg
triprpt.prg
tst.prg
upcen.prg
updemog.prg
upinter.prg
uppoints.prg
valcen.prg (obsolete/done (in db))
valdemog.prg (obsolete/done (in db))
valinter.prg (obsolete/done (in db))
AGE.PRG (obsolete/postgresql does it)
COMPARE.PRG
FEMFECUN.PRG
INSROW.PRG
INTEGRIT.PRG (Mostly done, needs review)
MERGEIP.PRG
minutes.PRG (obsolete/postgresql does it)
RNKDATE.PRG (done)
SMUSH.PRG
UNJULIAN.PRG (done)
UPCYCLE.PRG
UPMEMB.PRG (done)
UPSEQ.PRG
UPSUPERG.PRG
VALCYCLE.PRG (obsolete/done (in db))
VALMEMB.PRG (obsolete/done (in db))
ZEROS2.PRG
ZEROS.PRG

Offhand I can't remember what all of these do.  If you're not using
any let me know.

Plan is to do the ranker first.  That could take a week.  The
rest should be easy and take another week.  This is where there
begins to be something on the web site.

http://biology.duke.edu/babase/

Maybe we test by doing input work twice, in both the
new and old systems and then re-running the comparison test?

* Sometime after this we cut-over to the new system.

Week 4 (Oct 31):

The 'reminder' system that reports warning and errors and allows
them to be either ignored forever or put off until later.
The queries are done to generate the warnings/errors, probably,
but I need to do the rest.

Week 5+ (Nov 7):

Karl goes to Princeton for 2 weeks.

Document the new system.  This is a big-ish job (week?(s?)) involving 
going
over all the old docs and checking things.  While I'm at it
I'll be converting the format and putting it on the web/into pdf.

Work with staff.  Make 'standard' queries.  Maybe make some views.
Iron out problems.

Get rid of the babase_sandbox database and switch over to database
schemas -- put a sandbox area in all databases that students
et-al can make tables in that are kept separate from the babase
tables but where both sandbox and babase tables can be queried
together.

* Done with Babase 2.0!

Convert to text all the old foxpro datasets and other stuff used for 
input,
just to be able to keep them for the future.

Final archive of old foxpro system, with a separate archive of programs
for reference.

* Done with Foxpro!

Add new columns/tables and extend babase.

March boldly into the future.

The only thing I've left out here is the party, but maybe somebody
else can think of something.

Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                  -- Robert A. Heinlein