[saic] System error

[Daily SAIC News]

29 January 2006 ; Baltimore Sun
System error
http://www.baltimoresun.com/news/nationworld/bal-te.trailblazer29jan29,1,7228842.story

---

A Sun special report

The NSA has spent six years and hundreds of millions of dollars trying to
kick-start a program, intended to help protect the United States against
terrorism, that many experts say was doomed from the start.

A program that was supposed to help the National Security Agency pluck
out electronic data crucial to the nation's safety is not up and running
more than six years and $1.2 billion after it was launched, according to
current and former government officials.

The classified project, code-named Trailblazer, was promoted as the NSA's
state-of-the-art tool for sifting through an ocean of modern-day digital
communications and uncovering key nuggets to protect the nation against an
ever-changing collection of enemies.

Its main goal when it was launched in 1999 was to enable NSA analysts to
connect the 2 million bits of data the agency ingests every hour -- a task
that has grown increasingly complex with the advent of the Internet, cell
phones, and instant messaging -- and enable analysts to quickly pick out
the most important information.

The stakes could scarcely be higher.

A major failure leading up to Sept. 11, 2001, involved communications
intelligence, investigators found.More than 30 hints of the impending
attack had been collected in the previous three years but had sat,
unnoted, in the NSA's databases, according to a joint congressional
inquiry into pre-Sept. 11 intelligence operations.

The NSA initiative, which was designed to spot and analyze such hints, has
resulted in little more than detailed schematic drawings filling almost an
entire wall, according to intelligence experts familiar with the program.
After an estimated $1.2 billion in development costs, only a few isolated
analytical and technical tools have been produced, said an intelligence
expert with extensive knowledge of the program.

Trailblazer is "the biggest boondoggle going on now in the intelligence
community," said Matthew Aid, who has advised three recent federal
commissions and panels that investigated the Sept. 11 intelligence
failures.

Complex from the start - the initial Trailblazer plan called for more than
1,000 priority items - the project ballooned as it was passed through
three separate NSA divisions, each with its own priorities, former
intelligence officials said. And, they said, Trailblazer's overseers
lacked either the influence or the time to clearly define their goals and
keep the project on track.

When the agency's inspector general looked at the NSA's handling of the
project in its first three years, it found in a 2003 report "inadequate
management and oversight" of private contractors and overpayment for the
work that was done, according to a recently declassified version of the
report obtained by The Sun through a Freedom of Information Act request.

Meanwhile, Science Applications International Corp. (SAIC), the lead
contractor on the project, did not provide enough people with the
technical or management skills to produce such a sophisticated system,
according to industry and NSA experts familiar with Trailblazer. And, they
said, the company did not say no when the NSA made unrealistic demands.

The company was initially awarded $280 million in 2002 to begin
construction.

SAIC spokesman Jared Adams declined to comment, saying, "We have been
asked to defer all comment regarding the NSA Trailblazer contract to the
NSA."

The reporting in this article includes interviews conducted over the past
three months with 25 intelligence professionals, 13 of whom worked on or
had oversight of Trailblazer. Because the program is classified, most
would not allow their names to be used.

Although the Bush administration spent much of the past week defending the
NSA's eavesdropping work as vital to keeping Americans safe from
terrorism, virtually no attention has been paid to the agency's failure to
deliver the system the NSA said was key to fulfilling that mission.

That means the government has been standing by while the agency has been
gradually "going deaf" as unimportant communications drown out key pieces
of information, a government official with extensive knowledge of
Trailblazer told The Sun.

NSA spokesman Don Weber said the agency would have no response to requests
for comment.


Listening in

Based at Fort Meade in Anne Arundel County and with field offices around
the world, the NSA harvests virtually every form of electronic
communication - including phone calls, e-mails, video links and bank
transactions - through a vast array of satellites, clandestine posts at
U.S. embassies, ground-based listening stations, and military airplanes,
ships and submarines.

The information collected and culled by the agency's approximately 40,000
employees accounts for an estimated 75 percent of the president's daily
intelligence briefing, said Aid, an intelligence consultant who is writing
a multivolume history of the NSA.

But there are huge holes in the agency's information filter. As a result,
a congressional report on 9/11 intelligence failures found, "potentially
vital" information is lost, particularly with regard to terrorist groups.

That is what Trailblazer was designed to fix.

All digital communications trapped by the NSA are transmitted to the
agency's offices in computer codes of zeroes and ones. The sheer volume of
data being gathered is overwhelming the NSA's ability to digest it. And
that volume is growing every day with the advent of text messages,
hand-held computers such as the BlackBerry and phone conversations over
Internet lines.

The result is akin to looking for a needle in a haystack that doubles in
size every few months, said Aid, who has written extensively on
intelligence issues.

The agency has only blunt tools - largely based on the information's
origin or keywords linked to items of interest - to use in making
decisions about whether to keep captured data or discard it.

Intelligence experts familiar with the system said it is like deciding
whether to keep a piece of mail or throw it out based only on what is on
the outside of the envelope.

An estimated 95 percent of the information gathered is discarded without
being translated into an understandable form, said an intelligence expert
who has tracked the system for years. The remaining 5 percent, still in
the form of zeroes and ones, is turned into plain text or voice recordings
and routed to the appropriate division for analysis.

In each division, it might be run through software programs to identify
patterns or links with other data. But that is not guaranteed. Nor is
there a guarantee that a communication sent to a division dealing with
Latin America, for example, will ever be seen by an analyst tracking a
terrorist group that finances its activities through Latin American drug
smuggling.

NSA officials knew they needed to make changes in how they handled the
deluge of digital data and spent a year developing a broad concept for how
to do so.

As initially envisioned, said four intelligence experts with extensive
knowledge of the project, Trailblazer would have translated all of the
digital computer language (the zeroes and ones) into plain text or voice.
The data would have been analyzed to identify new patterns of activity or
connections among people whose communications are intercepted, and then
stored in an easily searchable database. Key communications would have
been automatically forwarded to the appropriate analysts, who for the
first time could have followed up with their own searches of the database.

To implement Trailblazer, the NSA would have vastly expanded its computing
hardware and software, and made revolutionary changes in the way huge
amounts of data are stored and retrieved.

But years after the initiative was launched, there was still no unanimity
within the agency on how to achieve those goals, or even on whether all of
them were necessary or possible, interviews and records show.

A December 2002 report by the House and Senate intelligence committees
investigating pre-Sept. 11 intelligence failures found that although
Trailblazer was "frequently cited" as the solution to many of the NSA's
information management problems, "implementation of those solutions is
three to five years away and confusion still exists as to what will
actually be provided by the program."

The unclassified report also noted that without Trailblazer the NSA
employees with whom committee investigators spoke knew of no "near-term
efforts to alleviate their current system's technical limitations."

Another division of the NSA had been working on a separate, less expensive
program, code-named Thinthread. In development before Trailblazer was
launched, Thinthread tried to accomplish a similar goal of separating the
important communications from the junk.

A classified report from the Pentagon in 2004 found that Thinthread was
more promising than Trailblazer and could be put to use faster, said an
intelligence expert who was briefed on its contents.

NSA managers disagreed with the Pentagon report's conclusions and canceled
Thinthread, said the expert briefed on the report's contents.

As a result, nearly 4 1/2 years after the Sept. 11 attacks, the NSA lacks
a system to comprehensively evaluate all of the communications collected
by its vast networks of high-tech ears.


Blazing trails

Trailblazer began as a signature program of Gen. Michael V. Hayden, who
was the NSA's director from March 1999 until last spring. Early on, former
officials familiar with the program said, it became clear to Hayden that
the agency, with its rich history of developing cutting-edge technology,
was falling behind the technology curve. He cast Trailblazer as the
agency's future.

A cerebral Air Force general with close-cropped hair and wire-rimmed
glasses, Hayden saw his tenure as a key opportunity to turn the agency
around. In November 1999, he made Trailblazer a centerpiece of his "100
Days of Change" agenda.

Presented nearly two years before the 9/11 attacks, former colleagues
noted, Hayden's plan was prescient.

"It was going to structure us to handle the digital revolution," said a
former intelligence official. And, the official recalled, it would start
by building on the agency's existing computer systems.

But two months after Trailblazer was launched, the agency's computers had
a 3 1/2 -day meltdown. Hayden later described the episode in a 60 Minutes
II interview as the agency's headquarters going "brain dead."

"It doesn't take a rocket scientist to know that an [information
technology] infrastructure that crashes ... is not going to be able to
handle Trailblazer applications," said a former intelligence official
familiar with the program.

Hayden declined, through a spokeswoman, to comment for this article and
referred questions to the NSA.

Several former intelligence and government oversight officials contend
that Trailblazer was doomed almost from its inception. The program "kicked
off with not a real great definition of what it was trying to achieve,"
said a government oversight official, recalling an initial briefing in
December 1999.

Trailblazer began with such a burst of energy that it skipped some crucial
first steps, said current and former government officials close to the
program.

For example, to make sense of the communications it pulled in, Trailblazer
needed a standard format for all data so that it could sort it properly,
much like the standards Google uses so that it can search different kinds
of information on the Internet. But intelligence officials said those
standards were never defined.

The agency also boxed itself in by underestimating how long it should keep
old data, a former national security official said. As a result, the
system was designed to discard information that could later prove useful,
particularly in an open-ended war on terror.

Such early errors were exacerbated by the Sept. 11 attacks, which prompted
Hayden to push for faster implementation, eliminating time for review and
corrections, a former intelligence official said. And Congress began
throwing money at Trailblazer, discouraging a more disciplined approach,
said a former government official with extensive knowledge of the program.


Monitoring project

While internal and external warnings that Trailblazer was going off course
were sounded, the extent of its problems gained little public attention
because the program was so secret and technical.

Since 1999, for example, more than 10 unclassified congressional reports
have pointed to "deficiencies" in NSA modernization efforts, but few
specifically pointed to problems with Trailblazer.

A 2003 NSA inspector general's report obtained by The Sun found that the
spy agency was unable to monitor the progress or the results of its early
Trailblazer contractors. Moreover, Inspector General Joel Brenner said
that his office could find no evidence of the program's specific
priorities, could not track the ways all of the money was being spent and
found that the NSA had overpaid some contract employees. The contracts
showed no limits on labor costs.

"These conditions are directly related to inadequate management and
oversight," Brenner's team said.

An intelligence expert who was briefed on a December 2004 report conducted
by the Pentagon's inspector general said the report found that Trailblazer
was not producing the system that had been promised and was unlikely to
produce it. The intelligence expert said the report suggested agencywide
management problems and recommended further investigations of the NSA's
overall acquisition and financial systems, at least one of which has
begun.

That report remains classified.

Shortly after the inspector general's review was completed, the NSA hired
IBM to take the lead on the project, said intelligence sources familiar
with the program.

Despite such warnings of problems, the Government Accountability Office,
the investigative arm of Congress that specializes in assessing program
management and government waste, has not looked into Trailblazer. Randolph
Hite, the GAO's director of information technology, said no one in
Congress has asked it to.

Former Sen. Bob Graham, a Florida Democrat who led the Senate Intelligence
Committee during and after the Sept. 11 attacks, said Congress has failed
to provide adequate oversight of the NSA.

"Most of the members don't have the background or the expertise to
understand very well an organization like the NSA," Graham said.

It did not help that Trailblazer, then less than two years old, was
faltering just as the country and Congress were attempting to cope with
the deaths of an estimated 3,000 people in the Sept. 11 attacks. Congress
was not interested in cutting any program related to fighting terrorism
for fear that it would be blamed if terrorists struck again.

>From that point, "our overwhelming focus was on trying to understand that
tragedy and the role the intelligence agencies had played," Graham said.
"Then, in 2002, summer and fall was the run-up to the war in Iraq, so our
attention was diverted."

That meant that on Capitol Hill, much of the oversight was left to the few
congressional staff members who understood the program. In July 2003, they
persuaded their bosses to send the NSA a no-confidence message about the
agency's ability to manage complex programs such as Trailblazer, said
congressional aides.

Congress took away the NSA's authority to sign big-ticket contracts
without getting permission from the Department of Defense. The Defense
Department continues to be responsible for approving the NSA's proposals
to pursue and pay for large programs.

But that didn't stop or fix the project. At the NSA, Trailblazer continued
to stumble along.

Rep. C.A. Dutch Ruppersberger, a Democrat from Maryland who sits on the
House Intelligence Committee and whose district includes NSA headquarters,
said abandoning the concept behind Trailblazer is not an option because
"our national security depends on it."

"There was congressional oversight, and that's one of the reasons this
program has been red-flagged as a program that needs work," he said. "The
conclusion we all had was there were mistakes made, but the concept has to
move forward for the sake of our national security."

Sen. John D. Rockefeller IV of West Virginia, who succeeded Graham as the
senior Democrat on the Senate Intelligence Committee, said the committee
has "worried about the direction of the Trailblazer program specifically
and NSA acquisition practices generally" over the past few years.

"We and the Armed Services Committee required detailed reports, withheld
money and ultimately removed the NSA's acquisition authority," Rockefeller
said. "I expect that [NSA Director Lt. Gen. Keith B.] Alexander will have
a major restructuring plan to present to the committee as part of the
request for fiscal year 2007."


Turf battles

To understand where Trailblazer ended up, it is helpful to understand the
internal politics at the time of its launch.

The beginning of the Trailblazer program in 1999 coincided with a major
NSA reorganization. Deep inside the spy agency, knowledgeable bureaucrats
rushed to get their programs redefined as part of a project favored by the
director and presumably immune from budget cuts, two former intelligence
officials said.

As a result, they said, Trailblazer's scope mushroomed in the first few
years.

Meanwhile, the project was passed among several divisions, including the
new Transformation Office, which was shut down after a year; the Signals
Intelligence Directorate, which housed many of the analysts; and the
Information Technology Directorate, which builds technology systems.

Each time Trailblazer was moved, its new leaders altered its design.

"Every year or so, their story would be somewhat different about what it
is, what it's going to accomplish and how it is going to be implemented,"
said a congressional aide who works with intelligence programs.

The program landed in the lap of William B. Black Jr., the agency's deputy
director.

Black had spent four decades at the NSA before leaving it in 1997 to join
SAIC, a San Diego-based contractor with strong ties to the agency. In
2000, Hayden called him back to become his top deputy and to take charge
of Trailblazer.

Two years later, the NSA awarded the prime contract to build Trailblazer
to SAIC, Black's former employer.

A careful bureaucrat who shies away from the media, Black was an expert at
navigating the agency's many fiefdoms and insisted that he make all key
decisions about Trailblazer, said intelligence officials with extensive
knowledge of the program. But they said Black had too much on his plate to
pay close attention to the program.

NSA spokesman Weber said Black was not available for an interview.

Lax internal oversight and shifting priorities quickly sent Trailblazer's
costs skyrocketing. In April 2005 Hayden testified before Congress that
the program, with publicly announced contracts then worth $500 million,
was "a couple to several hundred million" dollars over budget and behind
schedule.

Although he didn't provide details of the program's troubles, he
acknowledged in his testimony that getting the program off the ground "was
far more difficult than anyone anticipated."


New direction?

Five months into his tenure as NSA director, Alexander has been reviewing
Trailblazer, and he recently decided that he will try to revamp it rather
than scrap it, according to three intelligence experts familiar with the
program.

These officials and others knowledgeable about the program's history said
they were skeptical that Trailblazer could be fixed without starting from
scratch.

"Trailblazer is completely beyond fixing," said a former government
official who has tracked the program carefully. "Everybody who reviewed
Trailblazer after the first few months [of the program's launch] said it
was doomed or it should be scrapped."

Bobby Ray Inman, a former NSA director and a retired admiral, said there
needs to be some tolerance for altering the course of ambitious projects.

Several projects he considered successes, he said, were scaled-back
versions of an initial vision that "took us forward from where we were,
but it really didn't meet the aspirations of what we would have liked to
have had."

Alexander plans to hire a new executive to run the NSA's technology
programs, and Trailblazer will be one of this executive's top priorities,
said an intelligence consultant.

Alexander declined to comment for this article, but in August he told The
Sun that he would look to shift the agency's approach away from large
programs such as Trailblazer and toward smaller programs that build on one
another.

"I think the way to do it efficiently is smaller steps, more rapidly done,
rather than try to take one big jump and make it all the way across," he
said.

Those steps would involve significant changes in the way the NSA manages
data, including, he said, "how you handle data, how you visualize that
data and how we jump from Industrial Age analysis to the Information Age
analysis that our country needs."

Intelligence experts with extensive knowledge of the program said
Alexander is likely to salvage what he can from Trailblazer and largely
start over, casting it as a kind of "Trailblazer 2.0."

The country's new spymaster, Director of National Intelligence John D.
Negroponte, is taking on the job of connecting the technology systems of
all 15 intelligence agencies, and former intelligence officials said
Trailblazer's troubles should serve as a cautionary tale.

If Negroponte wants to learn the details, he won't have to go far. Since
last spring, his top deputy has been Hayden, the former NSA chief.