[Babase] Status report: old Babase more or less on new system
Karl O. Pinc
kop at meme.com
Wed Feb 8 23:10:40 EST 2006
Hi,
The old Babase data is on the new system in the new structure.
Please feel free to play with querying it but do not update it.
There are the following caveats:
Most data integrity checks are turned off. Any updates
will not be checked and will be overwritten anyhow
when we again run the conversion. It is probably not
a good idea to update anything.
Some data did not pass the remaining data integrity checks.
The data is loaded into the babase_copy database, not the
babase database, at present.
The CYCGAPS table has junk in it.
The new sexual cycle status tables, REPSTATS, CYCSTATS,
etc. are empty. I need to think/we need to talk about
how to get CYCGAPS right so that they can be properly
generated. I have at least a day's worth of work to do
once we figure this out. Is there a straightforward
list of gaps per group?
The state of data integrity is this:
The conversion errors are in
papio:/biology/groups/babase/errors
and
papio:/biology/groups/babase/loadscript
(per previous email)
The conversion errors reveal what did not convert.
Cathrine and I have previously gone over all the
MEMBERS data and are satisfied that it converted
completely. (Actually, CENSUS and DEMOG converted
and the new database successfully constructed the
MEMBERS table.)
As a completely independent check I ran queries
against each table in the old and new databases and
compared the result. The comparison results are
at:
papio:/biology/groups/babase/comparetables.out
These differences here should correspond with the
conversion errors, except that I have not bothered
to get the comparison to ignore slight differences
to the GROUPS table.
Because there are differences in the new and old database
structures the queries used in the database comparison
are slightly different from system to system.
The two sets of queries can be found at:
https://papio.biology.duke.edu/compare/
The file dumptext.prg is the Foxpro version.
The file dump_as_text is the Postgres version.
Some of the queries call functions, built-in on the
Postgres side and ones I wrote on the Foxpro side
to make the data come out comparably.
I'd like somebody to review the comparison output
and the queries to be sure that I did not forget
to compare a table or a column or otherwise do something
stupid. It is not really necessary to double
check the comparison output against the conversion
errors until we correct all the data and
are no longer getting errors.
Note that there are a few differences between
the old and new data that the comparison reveals.
The CENSUS table comes to mind but there may be
others. Somebody should look at this at some
point, although not necessarily right away.
(Could this have been due to the Foxpro system
being updated after I got the database files
but before the queries were run on Foxpro?
Probably they are due to conversion errors
and we should just re-evaluate after fixing
all the data problems the conversion reveals.)
Anybody who's interested might be able to
look at the two sets of queries and glean
something of the differences between the two
systems and the things that can be done with
the queries on the new system in comparison
with the old.
I should also note here that the web interface
to the database is very old and should be
upgraded to a newer version.
Cathrine and Leah. The master copy of the Foxpro
system at Princeton should now be updated with
the dumptext.prg program. Don't forget to
compile it. Cathrine, Leah says she will
help guide you.
Karl <kop at meme.com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
More information about the Babase
mailing list