Tax-filing software glitch cost $2.4 million to fix: report
OTTAWA, Ont. - A major glitch that shut down Canada Revenue Agency computers for more than a week, throwing 2,100 people out of work, cost taxpayers about $2.4 million to fix.
The government released the tally this week, three years after a simple software "patch" prevented thousands of Canadian taxpayers from filing their tax returns electronically.
The incident occurred at 3:20 a.m. on Sunday, March 4, 2007, when technicians installed a patch provided by consultant Computer Associates. The software fix, which was tested by agency officials before being installed, was intended to prevent large computer failures in the future.
But the repair itself created a major problem, and for the next 43 hours electronically filed tax returns did not land properly. Once the problem was discovered, more than a day after the installation of the patch, the agency halted all electronic filing of returns.
In early 2007, The Canadian Press asked for records of the incident under the Access to Information Act. The agency failed to deliver the material under the time limits set out in the law, and a complaint was filed with the information commissioner of Canada.
Canada Revenue Agency released some of the requested records, heavily censored, this week, more than 2.5 years after the legislated deadline.
At least 16,515 tax returns that were filed electronically during the so-called "corruption window" did not register properly in the computer systems, the documents show.
The agency sent 2,100 temporary workers home without pay across Canada, including 387 in Winnipeg and 315 in Shawinigan, Que. They were later recalled when the system was back in operation.
Technicians removed the patch soon after the glitch was discovered, but the computer did not restore properly. The agency needed about 10 days in total to get the systems back in service.
One post-mortem report says the software patch "was implemented and thoroughly tested in multiple test environments, prior to production with no adverse effects."
But agency officials apparently concluded the testing was not appropriate to the size of the tax system computers.
"The fact that this problem is only likely to occur in large organizations, and with large databases process(ing) large amounts of transactions, explains why it was not detected in our test environment," says one document.
The released material indicates that Computer Associates, now known as CA Canada, sent consultants to Ottawa to help repair the damage and did not charge the Canada Revenue Agency for those services.
A spokesman for the agency says a settlement was reached with the consultant over the $2.4 million in costs, which included $115,000 for public relations.
But Noel Carisse said the details remain confidential.
"A settlement was negotiated between the Canada Revenue Agency and Computer Associates, which included a non-disclosure agreement, which prohibits the disclosure of the terms of the settlement," Carisse said in an email.
A spokesperson for CA Canada, which remains a contractor for the agency, was not immediately available for comment.