CALPADS, the new comprehensive student data system on which huge hopes for school and student improvement are riding, is hobbled by serious problems.

Acting on a consultant’s report bluntly critical of state managers and of IBM, the system vendor, Superintendent of Public Instruction Jack O’Connell has shut CALPADS down for two months and ordered all efforts focused on fixing it. The hiatus will put data collection from the state’s 1,000 districts months, if not a year, behind schedule. (See correction: CALPADS is continuing to accept some data — dropouts, enrollments — while system is being overhauled.)

O’Connell had little choice but to act quickly. After studying the system for a month, Sabot Technologies of Folsom predicted a  “high probability of system failure should the project continue on the current path”  as a result of  “anomalies, errors and defects throughout” the system.

The California Longitudinal Pupil Achievement Data System will be the nation’s largest database of information on individual  students. By collecting information on demographics, student grades and test results, course enrollments and completion and discipline incidents, CALPADS can provide answers to important policy questions: who drops out and why, which school strategies work best with certain student groups, which university programs produce the best teachers, which high school indicators best predict success in college and the workplace.

But from the day it launched in October, local districts, which had to marry their own data systems to CALPADS,  have been frustrated by glitches and delays in feeding information.  They weren’t just whining. The Sabot Technologies report (only 24 pages and very readable) confirms that the system hasn’t operated “issue-free” for any length of time.

Costing tens of millions of dollars, CALPADS has been in development since November 2007. While too soon to place CALPADS in the pantheon of California computer fiascos, one can hear echoes of past problems – technically overmatched state management, for one – in the Sabot report.

On the bright side, the consultants found that the overall system architecture is sound. But little else was reassuring in the report.

Among the findings:

• There are quality defects in the system software, custom software, database and hardware architecture elements.

• IBM understaffed the project,  and its workers are “less experienced than needed for a project of this size,” contributing to a backlog of work.

• On the Department of Education side, there is a “distinct lack of technical leadership and engineering resources,” resulting in IBM making decisions on its own with “little transparency and oversight.” The contractor project manager is filling too many other roles; managing the project demands full attention.

• Engineering processes – patching software, releasing new software versions, responding to errors –  are “ad hoc, chaotic” and often undocumented. IBM “bears much responsibility” for the disorganized procedures, the report said, but  the state and the outside project managers failed to keep IBM to account.

Among the recommendations beyond an immediate full-scale review:

• Clarify roles among the project manager, the Department of Education and vendors, to stop confusion and inefficiency;

• Standardize procedures so that responses are less reactive and more thoughtful;

• Hire more oversight staff with technical expertise.

The  state will spend tens of millions of dollars on CALPADS between building the system and training district staff on using it (cost estimates differ). IBM and partners are to be paid $15 million for their work.

California has been counting on federal grants to finish the project.

By John Fensterwald on February 18th, 2010