Developer March

OFF THE PRESS

Vodafone Re-launches 15 Dollar Phone for Developing Countries Vodafone is tapping emerging markets and developing markets with two...

deCarta Location-enables Opera Browsers Location-enabled services have shown dramatic growth in the past twelve...

Aircel Deploys Oracle Solutions to Offer VAS and Increase Userbase to 50 Million Communications operators are working on creating competitive edge by...

WHITE PAPERS

A Business Case for Using Adobe Flex and the Adobe Flash Platform RIA combines the goodness of instant Web access to centralized data to create...

Cost Effective Defence-in-depth Security Controls and Solutions Information ranging from trade secrets to financial data to privacy related...

Boost Game Performance With Threads This white paper explains how to use functional decomposition to get better...

EXCLUSIVES

The Future of Distributed Computing Rests with Hadoop

Moore's law has finally hit the wall and CPU speeds have decreased in the last few years; the industry is reacting with hardware with more cores and software that can leverage 'grids' of distributed computing resources. Further, the assimilation of computing into our daily lives is enabling the generation of data at unprecedented rates. The amount of digital information churned out in 2011 is estimated to be 10 times that of what was produced in 2006; that is, 1800 exabytes. The rising number of web applications serving millions of Internet users and dealing with petabytes of data, the advent of cheap storage capacity resulting in a tremendous growth in data retention, and the availability of cheap resources to process that data have all reinstated the need for large-scale data processing.

Extracting information and intelligence from these data sets, commonly referred to as data analytics, is an important data intensive application stemming from this huge corpus of data. Data analytics is shown to be useful in several scenarios—analytics enable web data mining (for example, web indexing and search), enable extracting business intelligence (such as click stream analysis for increasing ad revenue), enable processing data sets from scientific studies and simulations (for research in natural language processing, seismic simulation and scene completion). Analyzing these large volumes of data demands a highly scalable solution.

Google's MapReduce is one popular approach that enables data analytics by parallely processing data, partitioned among large number of commodity machines. MapReduce can scale out to clusters of hundreds of commodity nodes as the data and processing demands scale, and can automatically handle failures in these large settings. Hadoop is a relatively new Java-based software framework that derives inspiration from Google’s MapReduce and FileSystem concepts.

Hadoop's Google influence doesn’t end there. Much like how the name Google does not mean anything, Hadoop's creator Doug Cutting, named it after his son's yellow stuffed elephant, afer discarding several others that were either already a web domain or trademarked. While the name was chosen for its originality and ease of pronunciation, Hadoop's basic purpose is to support applications that process vast amounts of data in a time efficient manner. Yahoo is Hadoop’s biggest supporter and it is extensively used in Yahoo’s search engine and advertising businesses.

In January 2010, Yahoo helped the Indian Institute of Technology in Bombay to set up a Hadoop cluster lab by donating a cluster of servers running the open-source Hadoop software. The cluster lab at Mumbai will help researchers at the institute study areas such as searching and ranking techniques, information extraction and natural language processing. Several others like Amazon, Adobe, AOL, Facebook, Google and Hulu use Hadoop clusters that scale up to thousands of computing nodes and petabytes of data.

Hadoop recently graduated to becoming a top-level project at the Apache Foundation. The Hadoop Distributed File System (HDFS) is a key component of Hadoop that is designed to store data on commodity hardware with high access bandwidth across the cluster.

HDFS = Hadoop Distributed File System

Hadoop's fault tolerant storage system called the Hadoop Distributed File System (HDFS), is capable to store huge amounts of information, scale up incrementally and survive the failure of significant parts of the storage infrastructure without losing data. An HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that each component has a non-trivial probability of failure means that some component of HDFS is always non-functional. Therefore, detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS.

The emphasis of high throughput instead of low latency makes HDFS appealing for batch processing. HDFS implements a simple data coherency model, write-once-read-many, that allows high throughput of data access. She portability of Hadoop makes integration easier with heterogeneous hardware and software platforms.

Hadoop at Great Indian Developer Summit

Hadoop is now considered the gold standard for the divide-and-conquer model of problem crunching. Matthew McCullough is an energetic 12 year veteran of enterprise software development and open source education. He is coming this summer to India's biggest polyglot conference and workshops for software professionals, Great Indian Developer Summit, to teach how to use the well-traveled Apache Hadoop framework.

Matthew is a member of the JCP and co-founder of Ambient Ideas. His experience includes successful J2EE, SOA, and Web Service implementations for real estate, financial management, and telecommunications firms, and several published open source libraries. His focus areas include Cloud Computing, Maven, iPhone, Distributed Version Control, and OSS Tools.

At GIDS 2010, Matthew will also be training IT professionals on other subjects such as Open Source Web Debugging Tools, covering the REST, HTML, SOAP, CSS, TCP, Filesystem and JavaScript facets of an app and look at utilities such as tcpdump, curl, Wireshark, JMeter, Firebug, JASH, Poster, SoapUI, Firediff, lsof, fs_usage, iwatch and more; iPhone application coding in Objective-C and integration with some favorite Java web service back-ends such as RESTful Grails; a whistle-stop tour of Maven 3.0, exploring the performance improvements, features that make debugging Maven issues easier, and changes to POMs that may require modifications to your build, but will result in more determinate build outputs

Matthew will also enable participants to leverage 100s of hours of research distilled into a 180 minute Cloud Computing boot camp on the Google App Engine, so they get bootstrapped with what cloud computing is and isn't, who the players are in this space, what unique features each offers, navigate through demos of building and deploying an app live to the Google App Engine, and the excellent tooling that the framework provides. Lastly, he will put a reality check on cloud computing, and GAE specifically, looking at pitfalls and gotchas.

Every year, GIDS is a game changer for several thousands of IT professionals, providing them with a competitive edge over their peers, enlightening them with bleeding-edge information most useful in their daily jobs, helping them network with world-class experts and visionaries, and providing them with a much needed thrust in their careers. Attend Great Indian Developer Summit to gain the information, education and solutions you seek. From post-conference workshops, breakout sessions by expert instructors, keynotes by industry heavyweights, enhanced networking opportunities, and more. Click here to register.

VIDEOS

PlayBook - Change the Game! Tablets are being billed to change the level of mobile communications. A new ground comes with its own challenges and oppurtunities. Andrew will talk about the BlackBerry PlayBook which...

PODCAST&VIDEOS

Interview with Sunil Rao, Head of Nokia Developer, India Sunil Rao is Head of Nokia Developer, India. He is responsible for evangelizing new technology platforms, strategy and growth opportunities with Nokia's third party developers to create a...

Value for Money System Integration is Teleca�s Strategy Teleca is a supplier of software services for all industries where mobile and wireless solutions are used. The company offers world class operations and execution capability, both on-site...

Security Architectures for the Rich Web While Web 2.0 technology and services have been common in the public sphere for a number of years, their uptake into the Enterprise has been relatively slow. Rich Internet Applications...

The Entire Game Should Move Onto the GPU, says Rev Lebaredian As the computing functionality and horsepower of GPUs has grown over the last few years, the role of the GPU is rapidly expanding to game tasks beyond rasterization-based graphics. Now...

Fermi Graphics Chip is Direct X 11 Done Right, says Nvidia's Ashu Rege and Keita Iida The computer game and hardware industry is advancing at a rapid pace, both in terms of technical innovation and sales. Nvidia is working towards expanding the market reach for games and...

Picking the Right Technology for Enterprise RIA We are entering an era of Rich Internet Applications (RIA), and many enterprise development managers are facing the dilemma - which way to go - remain with tried and true Java or .NET or...