At Demo, Hadoop inventor calls NSA snooping a cautionary tale for devs
Doug Cutting, who in 2004 developed the open-source implementation of the Map-Reduce framework, said big data analytics has opened the floodgates for capturing new consumer data as well as analyzing vast stores of historical information.
The NSA, for example, uses Hadoop for big data analytics related to its bulk data collection program. The agency has come under heavily criticism for its secretive, widespread collection of telephone and business records under the Patriot Act.
Cutting, who spoke at the DEMO Traction Enterprise conference here, said the NSA's first and biggest mistake was transparency.
"I think the NSA's recent struggles have been an example of an institution that wasn't well trusted because people didn't know a lot about what hey were doing and why they were doing it," he said. "We need to make sure that the people whose data we're collecting trust what we're doing with it.
"You need to build trust in from the start," he added.
Cummings, who is now chief architect for Hadoop services company Cloudera, said he has friends who refuse to use one web platform for all of their communication needs because they want to "confuse" vendors they don't trust.
For example, he said, people will keep their social and business calendars on one application and their email on another. While it makes their lives harder, it gives them a sense of security that all their information is not in one place, he said.
"To a degree that people are doing that, the vendors who provide these service have failed. They've failed the market if somebody doesn't trust them," Cumming said.
As the Web applications market moves forward, it needs to mimic what social networks have been doing by being more open about how they collect data and use it.
"We don't want people to degrade their use of our services and systems because they don't trust us," Cummings said..