HDFS: Useful Hadoop Admin Commands

HDFS is designed more for batch processing rather than interactive use by users. The emphasis is on high throughput of data access rather than low latency of data access.

Applications that run on HDFS have large data sets. A typical file in HDFS is gigabytes to terabytes in size. Thus, HDFS is tuned to support large files. 

HDFS categorises its data in files and directories.It provides a command line interface called the FS shell that lets the user interact with data in the HDFS and manage your hadoop cluster.

This article provides a quick handy reference to all commonly used hadoop fs commands that can be used to manage files on a Hadoop cluster.The syntax of the commands is similar to bash and csh.

Check more about -HDFSand MapReduce

File transferring from local to HDFS

Hadoop fs –put

File transferring from HDFS to Local

Hadoop fs –get

File transferring within the HDFS

Hadoop fs –cp

List of the File Display inside the HDFS

Hadoop fs –ls

Display the content in the File within HDFS

Hadoop fs –cat

Remove particular File in HDFS

Hadoop fs –rm

Remove File directory in HDFS

Hadoop fs –rmr

Display the File size

Hadoop fs –du <File_name>

Display the directory size

Hadoop fs –dus <directory_name>

Create new directory in HDFS

Hadoop fs –mkdir <directory_name>

Display the whole directory and its content in HDFS

Hadoop fs –lsr <directory_name>

Displays last kilobyte of the File

Hadoop fs –tail <File_name>

Display the overview of HDFS

Hadoop fsck

File transferring from local to HDFS

Hadoop fs -CopyFromLocal

File transferring from HDFS to Local

Hadoop fs –CopyToLocal

What is MapReduce in Big Data ?

MapReduce is pivotal to big data as it has allowed the processing of massive datasets, which the earlier preferred format for data storage and analysis, RBDMS was not capable of doing.

In terms of big data analytics, the open-source framework of Hadoop has been a game changer. It has enabled storage of large datasets going up to petabytes in a cluster format, and faster processing of this distributed data. An essential feature of Hadoop is its ability for parallel processing which is executed via MapReduce.

MapReduce is defined as the distributed data processing and querying programming engine that effectively splits and spreads around the necessary computation activities on a dataset across a wide range of servers which are known as data clusters.  A query that needs to run through a mega data set may take hours if situated in one computer server. This is however cut down to minutes when done in parallel over a spread of servers.

The term MapReduce refers to two critical tasks it handles on the Hadoop Distributed File System (HDFS) – the Map Job and the Reduce Job. The Map function takes the different input data elements available and processes them into an output data element, creating key value pairs.  The Reduce function aggregates outputs  created under the key value pairs, put them back together quickly and reliably in order to produce the required end-result.

Structurally, MapReduce has a single master Job Tracker and several slaves Task Tracker, one each per cluster. The master distributes and schedules the tasks to these slaves and keeps track of the assigned jobs, redoing any that fail.  The slave tracker ensures that the assigned task is executed and communicates with the master

There are number of benefits of MapReduce that has made it an important element of Hadoop.

• It allows developers to use any language like Java or C++ to write the applications, although Java is most preferred. 
• MapReduce can handle all forms of data whether structured or unstructured. 
• Another core feature of MapReduce is that it can easily run through petabytes of data stored in a data center due to its construct. 
• The MapReduce framework is also highly flexible in case of failures. If one dataset fails but is available in another machine, it can index and use the alternate location.  

Today, there are several additional data processing engines like Pig or Hive that can be used to extract data from a Hadoop framework. These eliminate some of the complexity of MapReduce and make it easier to generate insights.

Will discuss more about Map reduce in our upcoming post.

Learn more about Pig and Hive here

7 Top Big Data Tools for Enterprise Developers

Big Data is now a critical technology utilized for leveraging data to enable better decision-making. Developers today have a wide variety of choice in picking a tool for their needs – open source or proprietary.

A through assessment of the existing data structure and the business requirements is essential for developers to identify the right tool. Predominant data formats, existing database types, available budget and the desired output analytics are some important factors. A few of the top tools that can be considered by developers are given below:

1.   MongoDB

This is an open source platform that that is heavily document oriented allowing for full-fledged indexing. Users can index any attribute and can also scale the data horizontally. Its a cross-platform tool that allows developers great control over final results

Read more about this at Mongo DB-Features         

2.   SAP HANA

A proprietary platform from tech giant SAP, HANA offers in-memory data storage that speeds up data processing and delivers more insightful data. It is flexible in handling all forms of data including spatial data, graph data and text data, at real time, making it a powerful though an expensive tool for holistic data analytics.

3.   Google Charts

A free open source platform from Google, this has a wide range of capabilities to handle data off a website. Primarily used for visualization, it can be plugged into a website with a simple JavaScript code. Developers can use it to create dashboards, carry out data management tasks, pull data from an external database, among other tasks

4.   Hadoop

Hadoop is the Big Data tool that everyone has been talking about. An open-source framework, it can handle, store and process massive amounts of data. Due to a distributed computing model, the data processing is fast and powerful. The tool is highly flexible and scalable, making it an ideal choice to leverage Big Data analytics for enormous data sets

Read more about this at Hadoop-Features

5.   Spark

Spark is an highly popular open-source data processing platform and is said to be the most active Apache open source project under Big Data. It is an extremely fast – 100x faster than Hadoop- and flexible platform, enabling analyses all forms of structured and unstructured data. Its advantages include use of multiple languages and accessibility to other databases. 

6.   Splice Machine

Splice Machine is a SQL-on-Hadoop database that analyses data in real time. The tool allows developers to utilize standard SQL on it, giving it flexibility and making it easier to use.

Splice Machine which is also ACID-compliant product is available on a freemium basis and has a list price annual license fee of $5,000 per node.

7.   Splunk

It’s a popular Advanced IT Search Tool used by many companies which derives information from machine data. To make it more lucid, Splunk has the ability to search, monitor and analyze through all machine generated data such as log data generated by applications, servers, and network devices across an organization.

This product further indexes structured as well as unstructured data and helps in diagnosing the problems, making it easy for administrators, business analysts and managers to detect requisite information.

Read more at Splunk-Features

HIVE STORAGE FORMATS

In this post we will take a look on the different Storage File Formats and Record Formats in Hive

Before we move forward lets discuss for a split second about Apache Hive.

Apache Hive which is a data warehouse system for Hadoop facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems, first created at Facebook . Hive provide a means to project structure onto this data and query the data using a SQL-like language called HiveQL.…read more on Hive here

  

Among the different storage file formats that are used in hive, the default and simplest storage file format is the TEXTFILE.

TEXTFILE 

The data in a TEXTFILE is stored as plain text, one line per record. The TEXTFILE is very useful for sharing data with other tools and also when you want to manually edit the data in the file. However the TEXTFILE is less proficient when compared to the other formats.

SYNTAX :

CREATE TABLE TEXTFILE_TABLE (

COLUMN1 STRING,
COLUMN2 STRING,
COLUMN3 INT,
COLUMN4 INT

) STORED AS TEXTFILE;

SEQUENCE FILE

In sequence files the data is stored in a binary storage format consisting of binary key value pairs. A complete row is stored as single binary value. Sequence files are more compact than text and fit well the map-reduce output format. Sequence files do support block compression and can be compressed on value, or block level, to improve its IO profile further.

SEQUENCEFILE is a standard format that is supported by Hadoop itself and is good choice for Hive table storage especially when you want to integrate Hive with other techonolgies in the Hadoop ecosystem.

The USING  sequence  file keywords lets you create a sequence  File. Here is an example statement to create a table using sequence File:

 CREATE TABLE SEQUENCEFILE_TABLE (

COLUMN1 STRING,

COLUMN2 STRING,

COLUMN3 INT,

COLUMN4 INT

) STORED AS SEQUENCEFILE

Due to the complexity of reading sequence files, they are often only used for “in flight” data such as intermediate data storage used within a sequence of MapReduce jobs.

RCFILE OR RECORD COLUMNAR FILE

The RCFILE is one more file format that can be used with Hive. The RCFILE stores columns of a table in a record columnar format rather than row oriented fashion  and provides considerable compression and query performance benefits with highly efficient storage space utilization. Hive added the RCFile format in version 0.6.0.

RC file format is more useful when tables have large number of columns but only few columns are typically retrieved.

  

The RCFile combines multiple functions to provide the following features

•  Fast data storing
•  Improved query processing,
• Optimized storage space utilization
• Dynamic data access patterns.

SYNTAX:

CREATE TABLE RCFILE_TABLE (

COLUMN1 STRING,

COLUMN2 STRING,

COLUMN3 INT,

COLUMN4 INT ) STORED AS RCFILE;

Compressed RCFile reduces the IO and storage significantly over text, sequence file, and row formats. Compression on a column base is more efficient here since it can take advantage of similarity of the data in a column.

ORC FILE OR OPTIMIZED ROW COLUMNAR FILE

ORCFILE stands for Optimized Row Columnar File and it’s a new Hive File Format that was created to provide many advantages over the RCFILE format while processing data. The ORC File format comes with the Hive 0.11 version and cannot be used with previous versions.

Lightweight indexes are included with ORC file to improve the performance.

Also it uses specific encoders for different column data types to improve compression further, e.g. variable length compression on integers 

ORC stores collections of rows in one file and within the collection the row data is stored in a columnar format allowing parallel processing of row collections across a cluster.

ORC files compress better than RC files, enabling faster queries. To use it just add STORED AS orc to the end of your create table statements like this:

CREATE TABLE mytable (

COLUMN1 STRING,

COLUMN2 STRING,

COLUMN3 INT,

COLUMN4 INT

) STORED AS orc;

If you like this post, please share it on google by clicking on the Google +1 button.

What is Cloudera Impala ? Impala vs Hive

Cloudera Impala is an open source, and one of the leading analytic massively parallelprocessing (MPP) SQL query engine that runs natively in Apache Hadoop. Cloudera Impala project was announced in October 2012 and after successful beta test distribution and became generally available in May 2013.Its preferred users are analysts doing ad-hoc queries over the massive data sets stored in Hadoop.

The main feature of Impala is that with Impala we can run low-latency Adhoc SQL queries directly on the data stored in a cluster, stored either in unstructured flat files in the file system, or in structured HBase tables without requiring data movement or transformation. Performance is increased due to the fact that we need not migrate data sets to dedicated processing systems or convert data formats prior to analysis.

Another important feature of Impala is that it is workable to the data formats metadata, security and resource management frameworks used by Map Reduce, Apache Hive, Apache Pig and other components of the Hadoop stack.

Impala also supports all Hadoop file formats, including new format Apache Parquet. Apache Parquet is a columnar storage format for the Hadoop ecosystem created with advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model, or programming language.

Impala queries are executed as follows:

• Queries are submitted using Impala-shell command-line tool, or from a business application through an ODBC or JDBC driver.
• Impala distributed query engine builds and distributes the query plan across the cluster.
• It runs separate Impala Daemon (impalad) which runs on data nodes and responds to impala shell. These daemons can return data quickly without having to go through a whole Map/Reduce job.
• Impalad is a process that runs on designated nodes in the cluster. It coordinates and runs queries.

Comparison With Hive 

When we compare to Hive and MapReduce ,both optimized for long running batch-oriented tasks such as ETL(Read more:What is ETL), Impala is more compatible for running interactive analytical SQL queries over small amounts of a huge data. What makes it different form HIVE is that Impala does not rely on Map Reduce, it avoids the start-up overhead of Map Reduce jobs and instead uses its own t’s own set of execution daemons which need to be installed alongside your data nodes. 

Hive in Hadoop ecosystem is intended for a data warehouse system to support with easy data aggregations, adhoc queries over large datasets which are stored in Hadoop HDFS file systems whereas Cloudera Impala is a query engine for data stored in HDFS and HBase. 

Both Hive and Impala supports the existing HiveQL(a SQL-like language-Read more on :Apache Hive ).

Because Impala and Hive share the same metastore database and their tables are often used interchangeably. This cross-compatibility applies to Hive tables that use Impala-compatible types for all columns.

Partitions in Impala 

As in large scale Data warehouse how we make use of partitioned tables (Read more on: Partitions in Oracle ) to speed up queries, the same way in Impala we make use of Partitioned tables. Data is partitioned based on values in one column and instead of looking up one row at a time from widely scattered items, the rows with identical partition keys are physically grouped together. Impala also takes advantage of the partitioning present in Hive tables.

Cloudera Impala makes use of the following two technologies

• Columnar Storage:  Since data stored in columnar fashion it gives high compression ratio and efficient scanning.
• Tree Architecture: The architecture forms a massively parallel distributed multi-level serving tree for pushing down a query to the tree and then aggregating the results from the leaves.

Impala provides the following benefits:

• Efficient resource usage: Impala can handle concurrent client requests in shared workload environment. Each Impala daemon can handle multiple concurrent client requests
• Impala doesn't provide fault-tolerance compared to Hive. Just in case the node fails in the middle of processing, the whole query has to be re-run. But Impala has the advantage that even if node fails and we start over, its total runtime is so fast that it will accomplish for the time loss.
• Time savings because you do not have to move around data and Impala does not write the intermediate results to disk.
• Supports Hadoop Security (Kerberos authentication) and role-based authorization through the Apache Sentry project.
• Far-reaching accessibility of Hadoop data to the business community.
• More complete analysis of full raw and historical data, without information loss from aggregations or conforming to fixed schemas.

If you like this post, please share it on google by clicking on the Google +1 button.

Please go through our latest post TOP 6 BIG DATA TRENDS IN THE NEAR FUTURE

Google BigQuery- An externalized version of Dremel

So What is Google Big Query?? Its powerful Big Data analytics platform used by all types of organizations to run SQL-like queries against multiple terabytes of data in a matter of seconds. With this cloud based interactive query service we can handle web-sized amounts of data at blazing fast speed. 


Big Query (released in 2010)is actually the external or public implementation of one of the Google’s core technologies so-called Dremel .Big Query provides the features available in Dermel to third party conserving its unparalleled query performance. Both in fact share the same underlying architecture and performance characteristics. 

Big Query release made it possible to utilize the power of Dremel and to take advantage of Google’s massive computational infrastructure.

Let’s take a deeper look into power of Dremel… It is a query service that allows you to run SQL-like queries against very, very large data sets and get accurate results in mere seconds. You just need a basic knowledge of SQL to query extremely large datasets in an ad hoc manner.

Dermel runs through tens of thousands of servers simultaneously and makes it easy to analyse large amount of data such as a collection of web documents or a library of digital books or even the data describing millions of spam messages.

“According to Google’s paper, this has been used inside Google since 2006, with “thousands” of Googlers using it to analyse everything from the software crash reports for various Google services to the behavior of disks inside the company’s data centers”


The two core technologies that makes Dremel and BigQuery so fast is the Tree Architecture of Dremel And that the Data is stored in a Columnar Storage fashion in so doing, it gives very high compression ratio and scan throughput. 

So how to use data in Big Query or how to import data into Big Query:

• Upload your data to Google Cloud Storage
• Import the files to Big Query. Executed using command-line tool, Web UI or API, which can typically import roughly 100 GB within a half hour.

Other Important Features of Google Big Query:

• BigQuery is designed to handle structured data using SQL. Apart from SQL queries we can easily read and write data in Big Query via Cloud Dataflow, Spark, and Hadoop

• BigQuery provides extremely high cost effectiveness and full-scan performance for ad hoc queries and cost effectiveness compared to traditional data warehouse solutions and appliances.

• BigQuery is the best choice for ad hoc OLAP/BI queries that require results as fast as possible.

• BigQuery requires no capacity planning, provisioning, 24x7 monitoring or operations, nor does it require manual security patch updates. You simply upload datasets to Google Cloud Storage of your account, import them into Big Query, and let Google’s experts manage the rest.

If you like this post, please share it on google by clicking on the Google +1 button.

Please go through our latest post TOP 6 BIG DATA TRENDS IN THE NEAR FUTURE

INTERNET OF THINGS (IOT)-MEANING AND EXAMPLES

The Internet of Things (IoT)  is a fast emerging network of IP-connected  devices and the communication (transfer of data) that occurs amid these devices without any human interface. IoT devices utilizes wireless sensor networks, control systems, micro-electromechanical systems (MEMS) and the Internet to interact with external environment.

Kevin Ashton, cofounder and executive director of the Auto-ID Center at MIT, first mentioned the Internet of Things: "All the information available on internet was captured or created by human beings(by typing, pressing a record button, taking a digital picture or scanning a bar code) with limited accuracy and attention.Instead if we could use computers to track things then we could know when things needed replacing, repairing or recalling and whether they were fresh or past their best".

Examples that come under  Internet of Things consist of  thermostats, cars, electronic appliances, lights in household and commercial environments, alarm clocks, speaker systems, vending machines and more.Few other scenarios where IoT comes into picture are

• Automobile with built in sensors to alert the driver in case of any concern.Its possible to detect when components like batteries will fail and replacement is needed.
• Printers detects its low ink level and order for replacement.
• Stock-out prevention through connected and intelligent supply chains.
• A person with a heart monitor implant.

Objects that are part of IoT becomes connected to the surrounding objects and helps in creating creating a network of connections between secluded data sources.

Important thing here is the handing over of IP address for these IoT devices.According to Steve Leibson, who identifies himself as “occasional docent at the Computer History Museum,” the address space expansion means that we could “assign an IPV6 address to every atom on the surface of the earth, and still have enough addresses left to do another 100+ earths.”

SECURITY in the IoT:

As the number of IoT devices augments the security concerns for this also increases.Studies disclose that by 2020 there will be around 50 billion connected devices. Hackers can make use your data and can change your requirement.Consequently its needed that we look after the IoT devices against malware, spyware, worms, and trojans.

If you like this post, please share it on google by clicking on the Google +1 button.

Please go through our latest post TOP 6 BIG DATA TRENDS IN THE NEAR FUTURE