UPDATE: Computer glitch affects 24 state agencies

By Nicole Bell - bio | email
Posted by Terry Alexander - email

UPDATE: See the schedule of updates released by VITA at the bottom of this story

RICHMOND, VA (WWBT) – Hundreds of DMV customers were turned away on Thursday due to a glitch in the agency's computer system. The malfunction is part of a larger problem that's affecting 24 state agencies across the state.

The two agencies most impacted are the Department of Motor Vehicles and the Department of Taxation. Customers who showed up, specifically for a driver's license transaction at DMV's across the commonwealth were told to come back another day.

Zakia Hunter considers herself lucky. She was able to complete her business at the DMV with ease.

"I came to renew my vehicle registration. I got my sticker renewed in less than 30 minutes", said Hunter.

More than 1800 DMV customer, statewide, were turned away on Thursday.

"Driver's license transactions aren't being processed today. We can't capture images on the server that's the nitty-gritty of the problem," said DMV spokesperson, Melanie Stokes.

It's something Shelly Rainey and her family call frustrating. Rainey was hoping to get a duplicate of her son's license.

"We thought we'd be in and out to get the duplicate, and it hasn't worked out that way," said Rainey.

The computer snafu at DMV is part of a statewide problem. The Virginia Information Technologies Agency or VITA provides computing services for some state agencies.

VITA is experiencing a network issue with its server. As a result, 24 state agencies are being impacted -- including the Employment Commission, the Department of Emergency Management, VDOT, and Alcoholic Beverage Control. VITA says the problem started Wednesday afternoon.

Back at the DMV customers like Jessica Parker hope for a solution sooner than later.

"I guess I'll try again next week," said Parker.

VITA says it's unclear when the problem will be fixed. DMV customers who need a driver's license transaction are encouraged to contact the agency before showing up.

If you simply need to renew your license you can take care of that online at www.dmvNOW.com, or use the automated telephone service at 1-888-337-4782. VITA says rapid response workers are doing everything they can to correct the problem as quickly as possible.

Update on storage outage 11 a.m., Friday, Aug. 27, 2010

·       Maintenance and repair work proceeded as expected overnight. Applications, servers and the damaged storage system were taken down as planned.

·       The storage system has been repaired.

·       More than 60 percent of the servers attached to that storage system have been brought back up and are operational. Agencies are testing those servers and related applications.

·       Work continues to restore the remaining servers.

·       The system is not yet 100 percent operational. We will be brining services up throughout the day. Some data must be restored.

·       Unfortunately, DMV still cannot process driver's licenses at its customer centers. Please check the DMV website for details on services available at the customer centers and online. Some other agencies continue to be impacted.

Update on storage outage 3 p.m., Thursday, Aug. 26, 2010

·       While work continued today (Thursday, August 26, 2010) on repairing the faulty networked storage system, information technology operations continued in a degraded mode impacting 24 state agencies.

·       The storage provider, EMC, determined that the best course of action is to perform an extensive maintenance and repair process. VITA and Northrop Grumman, in consultation, have determined this is the best way to proceed.

·       That process of maintenance and repair will begin at 5:30 p.m. today.

·       That process will impact the original 24 agencies. Impact to additional agencies still is being determined. We are aggressively notifying agencies of these plans so that they can plan accordingly.

·       VITA, Northrop Grumman and EMC will coordinate efforts to execute a plan to replace the failed components and any others that appear suspect. This process involves shutting down applications, bringing down servers and bringing down the storage system, and restoring all three in reverse order. Applications will be brought back online during the early morning hours tomorrow.

·       Afterwards, corruption of some data is expected. VITA and Northrop Grumman will work directly with agencies tomorrow for remediation and to restore data from backups.

·       This is the best course of action to remedy the current situation and bring all applications back on line as expeditiously as possible.

·       The SAN is a high-performance, highly reliable unit. EMC representatives report it is highly unusual to encounter a dual failure of memory boards in a networked storage system. This is a unique issue that requires this extensive maintenance and repair process.

·       Special thank you to citizens and state employees impacted for their patience and understanding during this challenge.

Storage outage update 10 a.m., Thursday, Aug. 26

·       A storage area network (SAN) failed. Redundancy also failed. This hardware failure occurred shortly before 3 p.m. Wednesday.

·       Servers attached to this storage network could not access data.

·       VITA and Northrop Grumman activated the rapid response team and began work with the appropriate vendors to restore service. Teams have worked through the night.

·       Not all systems at a given agency are down. Possible impacts include databases, applications and websites

·       Approximately two dozen agencies are impacted. For security reasons and because the degree of severity varies at the impacted agencies, we are not providing the names of those agencies.

·       Such failures occur with information technology systems, the same as they do with power grids. We work to avoid such failures, minimize the impact and restore service as quickly as possible. And, we are doing just that now.

·       The work we have done in recent years to improve the information technology for the state has made our systems more secure and reliable. When something like this does happens, we have processes in place and dedicated, knowledgeable staff at the agencies, VITA and Northrop Grumman to respond appropriately.

Copyright 2010 WWBT NBC12. All rights reserved.