HDFS: Useful Hadoop Admin Commands

HDFS is designed more for batch processing rather than interactive use by users. The emphasis is on high throughput of data access rather than low latency of data access.

Applications that run on HDFS have large data sets. A typical file in HDFS is gigabytes to terabytes in size. Thus, HDFS is tuned to support large files. 

HDFS categorises its data in files and directories.It provides a command line interface called the FS shell that lets the user interact with data in the HDFS and manage your hadoop cluster.

This article provides a quick handy reference to all commonly used hadoop fs commands that can be used to manage files on a Hadoop cluster.The syntax of the commands is similar to bash and csh.

Check more about -HDFSand MapReduce

File transferring from local to HDFS

Hadoop fs –put

File transferring from HDFS to Local

Hadoop fs –get

File transferring within the HDFS

Hadoop fs –cp

List of the File Display inside the HDFS

Hadoop fs –ls

Display the content in the File within HDFS

Hadoop fs –cat

Remove particular File in HDFS

Hadoop fs –rm

Remove File directory in HDFS

Hadoop fs –rmr

Display the File size

Hadoop fs –du <File_name>

Display the directory size

Hadoop fs –dus <directory_name>

Create new directory in HDFS

Hadoop fs –mkdir <directory_name>

Display the whole directory and its content in HDFS

Hadoop fs –lsr <directory_name>

Displays last kilobyte of the File

Hadoop fs –tail <File_name>

Display the overview of HDFS

Hadoop fsck

File transferring from local to HDFS

Hadoop fs -CopyFromLocal

File transferring from HDFS to Local

Hadoop fs –CopyToLocal

Overview Of Scala Programming Language- How To Install Scala ?

SCALA (Scalable Language)  is a modern programming language created by Martin Odersky , influenced by Java, Ruby, Smalltalk and others.

Scala smoothly and effortlessly integrates object-oriented and functional programming .It supports Java-like superior coding style and simultaneously supports a functional style.

Combining the strengths of the Functional and Imperative programming models, Scala is a great tool for building highly concurrent applications without surrendering the advantages of an OO methodology. 

Features of Scala Programming Language

• Scala code runs on the JVM and allows you to use the affluence of Java libraries that have been developed over the years.
• Features in object oriented programming (OOP) language such as Classes and inheritance are fully supported in Scala. Some other features in Scala are traits, objects and case classes.
• Every variable here is an object (Variable definitions start with var ) and every “operator” is a method. It’s also has functional programming (FP) language, so you can pass functions (Function definitions start with def.) around as variables.
• Scala has advanced language features and rich Java integration.You can write your Scala code using OOP, FP, or both.  
• Scala has expressive syntax and static typing

Installing Scala Software

For Installing Scala On Unix systems download the software from the Scala download page to a directory on your computer like $HOME/scala, and then add these lines to your $HOME/.bash_profile file.

export SCALA_HOME=/Users/Al/scala

PATH=$PATH:/Users/Al/scala/bin

To Install Scala in Microsoft Windows you can follow equivalent process. Check the Scala download page for more information.

Books to Read on Scala

1.          Programming in Scala: A comprehensive Step-by-Step Scala Programming Guide by MartinOdersky, Lex Spoon, Bill Venners
2.          Scala for the Impatient by Cay Hortsmann
3.          Scala in Depth by Joshua D Suereth
4.          Introduction to the Art of Programming Using Scala by Mark Lewis
5.          Atomic Scala by Eckel and Marsh
6.          Functional Programming in Scala by Paul Chiusano and Rúnar Bjarnason

The Rise and Rule of Cassandra

The world of database is dark and full of terrors. We spent decades working happily upon relational databases, until realizing one day that relations are not enough. This paved way for NoSQL databases, or alternatively, any database that do not use tables. These new databases were shiny and cool, but could not match the massive power that Oracle and SQLServer wielded. This changed with the arrival of Cassandra.

When two scientists in Facebook in 2008 decided to build a database different from other NoSQL databases, they only wanted to have their own database. But little did they know of the impact they would have on the industry.

Cassandra was introduced with a sole objective: to solve the crisis of scalability. And it managed that beautifully. In fact, it has consistently been cited as the only NoSQL database that can take as many machines as could be added to it, without breaking a sweat. In 2012, a group of researchers from University of Toronto declared that as far as scalability goes, there is no match for Cassandra. But this has not been the only reason behind the vast popularity of Cassandra.

Cassandra prides itself as having no "single-point of failure", which implies that there is no single component whose failure can shut down the whole database. In a world where transactions are carried out every second, this feature of robustness is of vital importance. Many talk about decentralization, but nobody does it better than Cassandra.

But having a couple of advantages does not make you better, not in a world of ruthless competition. MongoDB, Redis and others would not be amused by a database who would take away their market with a couple of features. This is why Cassandra tried to achieve perfection. Its fault-tolerant and decentralized nature makes it extremely durable, thus being the perfect choice for those organizations who cannot afford to lose even an ounce of data. The throughput increase is linear with respect to growth in size, which makes it extremely desirable for databases which are growing constantly. After providing all these features, it is not a surprise that Cassandra is trusted by some of the biggest names in the industry, including CERN, eBay, Instagram, GoDaddy, Netflix and Reddit. In fact, Apple's deployment of Cassandra stores a whopping 10 PB of data across 75000 (and growing) nodes. Cassandra can give lessons on scalability to every other non-relational database.

That said, Cassandra is still not the most popular database around; it is not even the most popular NoSQL database right now. There are few inherent flaws that Cassandra needs to fix, including simplified deployment, simplified operational maintenance and an improved web interface, among other things. There is still the issue of low predictability of performance (which was partially reduced, but never solved) and the complexity of APIs in the client libraries which is nothing but unnecessary. But Cassandra is growing strong, and the time is not far when it will be a common name among all DB designers.

What is MapReduce in Big Data ?

MapReduce is pivotal to big data as it has allowed the processing of massive datasets, which the earlier preferred format for data storage and analysis, RBDMS was not capable of doing.

In terms of big data analytics, the open-source framework of Hadoop has been a game changer. It has enabled storage of large datasets going up to petabytes in a cluster format, and faster processing of this distributed data. An essential feature of Hadoop is its ability for parallel processing which is executed via MapReduce.

MapReduce is defined as the distributed data processing and querying programming engine that effectively splits and spreads around the necessary computation activities on a dataset across a wide range of servers which are known as data clusters.  A query that needs to run through a mega data set may take hours if situated in one computer server. This is however cut down to minutes when done in parallel over a spread of servers.

The term MapReduce refers to two critical tasks it handles on the Hadoop Distributed File System (HDFS) – the Map Job and the Reduce Job. The Map function takes the different input data elements available and processes them into an output data element, creating key value pairs.  The Reduce function aggregates outputs  created under the key value pairs, put them back together quickly and reliably in order to produce the required end-result.

Structurally, MapReduce has a single master Job Tracker and several slaves Task Tracker, one each per cluster. The master distributes and schedules the tasks to these slaves and keeps track of the assigned jobs, redoing any that fail.  The slave tracker ensures that the assigned task is executed and communicates with the master

There are number of benefits of MapReduce that has made it an important element of Hadoop.

• It allows developers to use any language like Java or C++ to write the applications, although Java is most preferred. 
• MapReduce can handle all forms of data whether structured or unstructured. 
• Another core feature of MapReduce is that it can easily run through petabytes of data stored in a data center due to its construct. 
• The MapReduce framework is also highly flexible in case of failures. If one dataset fails but is available in another machine, it can index and use the alternate location.  

Today, there are several additional data processing engines like Pig or Hive that can be used to extract data from a Hadoop framework. These eliminate some of the complexity of MapReduce and make it easier to generate insights.

Will discuss more about Map reduce in our upcoming post.

Learn more about Pig and Hive here

Overview of Mongo DB 3.4 : New Features

Mongo DB has been wildly popular ever since its introduction for plenty of reasons. The biggest one was because it got rid of the Object-Relation Mapping to a large extent, which had been the source of trouble of programmers for years. Even today, it is the 5th most popular database. However, the graph of popularity of Mongo DB decreased somewhat over the years, due to introduction of more advanced and simplified NoSQL databases. This might change with the release of Mongo DB 3.4, released late last year. According to the company, they seek to attain a "digital transformation" with this release.

The clear message that the company gave with this release was that it is aiming to simplify the life of large enterprises that have depended upon Mongo DB for long now. Like Python, Mongo DB is aiming to evolve so that it alone suffices for tasks that earlier required multiple technologies. Since we have seen this formula succeeding more than once, we have to admit this is a very smart move from the company.

Graph support was the need of the hour in Mongo DB for quite some time now. Taking more than 3 years to become a reality, it is arguably the biggest addition in the new version. While it does not seem to pose any threat to established graph databases like Neo4J, the graph support is sure to simplify things for its existing users. This feature is sure to have large impact, as it will facilitate companies to explore hitherto doubted avenues like Deep Analytics, Internet of Things and Artificial Intelligence. This would be further aided by Atlas, Mongo DB's database cloud service released earlier last year.

Ecommerce websites working upon Mongo DB had toiled hard for long to provide decent search functionality to its customers. This ends with the faceted navigation feature, which uses filters to narrow down the query results. This ensures faster and more relevant search results. Also, a read-only mode was introduced that could expose the information of an application while preventing any modification. Another huge feature was the creation of Geo-distributed Mongo DB zones, which deals with the problem of data sovereignty and solves it by providing tagging via a higher abstraction of “zones”.

The release also had few things in store for the regular users. The new SQL interface is sure to greatly ease things for the users who have struggled for long to import their SQL code into Mongo. Mongo DB also introduced the ($switch) operator, which greatly simplifies complex branching, while making it more readable. Like the popular "switch" expression, it tests a number of cases, executing only the one that turns out to be true. Another addition was the ($reduce) operator, that could reduce the results of multiple arrays into a single expression.

Apart from this, there has been a whole array of other additions, whose actual importance would only be realized in the long run. This includes elastic clustering, tunable consistency and enhanced DBA.

Overall, this release has been quite impressive and an instant success. Mongo DB has made its intention very clear: It is here to stay and win. With this, other NoSQL providers like Redis and Cassandra as well as established SQL players like MySQL and Oracle will have to up their game.

Pig vs Hive: Main differences between Apache Pig and Hive

Delving into the big data and extracting insights from it requires robust tools that allow flexibility in data management and querying – filtering, aggregating, and analyses. Typically, MapReduce code is leveraged to do this but the complexity involved in writing intricate Java code to prepare MapReduce scripts led to new languages being created that allowed users to access datasets with more ease.

Pig was created by researchers at Yahoo, and has the flexibility of multiple query approach. Although somewhat similar to SQL the traditional language for data analysis in some ways, doesn’t have its declarative nature and has limitations like- being dependent on relational database schemas. Pig is more of a programming language, and is often referred to as an abstraction of the complicated syntax of Java programming required for MapReduce. Pig has has different semantics than Hive and Sql.

Hive (invented at Facebook) on the other hand is highly similar to SQL, as it uses almost the same commands for data manipulation, making particularly suitable for those experienced in use of SQL.

These two components of the Hadoop ecosystem work atop Hadoop. The goal of both these tools is to make it easier to interact with massive datasets within Hadoop without having to write out complex MapReduce code.

Understanding the differences between Pig and Hive  

There are several differentiating elements between the two languages, and big data users need to appreciate these differences to make use of the right tool:

•  As Hive adopts SQL-based declarative approach it is often preferred for structured data especially historical data. It is therefore often referred to as a data warehouse platform.

Pig on the other hand uses Procedural Data Flow Language and is preferred for semi- structured, unstructured or decentralized data. The flexibility of Pig allows better construction of data flows and its feature of self-optimization results in lesser number of data scans.

• Hive use distinct query language called HQL whereas Pig use their own language called piglatin (procedural language).
• Partitioning can be done using HIVE whereas it’s not possible in in PIG
• In terms of practical usage, Hive is preferred for reporting and operates on the server side of a cluster while Pig is great for writing programs and operates on the client side.
• Given its characteristics, Pig is typically used by researchers and programmers but Hive is preferred by data scientists who work on large quantitative datasets.
• Hive usually executes quickly but loads slowly whereas Pig loads faster and more effectively.

Adopting a standard approach to big data analytics would hamper benefits from it.  Both Pig and Hive have their own advantages that make them apt for some situations but not in others. Analysts must carefully examine the insight requirements before deciding on the tool to use. 

How and Why To Bridge between SQL and NoSQL

SQL have for long now been the synonym of "database" for us. For any sort of data management, SQL had been our instinctive choice. However, the past decade saw the emergence of NoSQL which gave rise to a fierce competition of preferences.

  

What haunts the mind of every aspiring database developer today is the question of choice: To SQL or NoSQL. We want to keep in touch with the latest trends in the technology, but don't want the established technologies to slip away either. However, the most basic point that most people seem to miss is this: SQL and NoSQL are not competitors, and most certainly not antonyms of each other.

SQL or Structured Query Language is the most standard concept of database management systems today. SQL considers data to be stored in the form of tables called Relations, that consist of tuples and attributes. While this concept had been a hugely successful improvement over the data-storage systems present at that time, like flat files, things have changed today.

NoSQL came as a breath of fresh air in an industry that was rapidly changing. The world is going digital, and the digital world is messy. We can never predict the volume, variety or velocity of incoming data. The data, apart from being unpredictable, is also unstructured. Since relational databases are not inherently adept to handle them, something else was required. At the same time, distributed computing is all the rage today, because most businesses are moving towards the cloud. The expansion of relational databases cannot keep up with the pace; thus, NoSQL entered into the scene.

Why to migrate from SQL to NoSQL

Strictly speaking, NoSQL aims to do what SQL cannot. It is not based on relations and it may sometimes even fail to follow the ACID properties! But unlike what you have been taught, ACID properties, though really useful, are not the ultimate necessity. The ultimate necessity is fault tolerance, and NoSQL manages to achieve that anyway.

NoSQL cannot be defined in a single line, as there is no single definition. While all SQL-based databases follow strict guidelines that adhere to SQL-standards, NoSQL gives the databases a free rein. With so many lacks of standards, one might wonder: Are the reasons enough to migrate to NoSQL?

Yes, because we have only touched the crux of the importance of NoSQL in modern world. The two biggest reasons why NoSQL trumps over SQL are agility and scalability.

With the rapid changes that occur daily in the industry, being agile is the only way to survive. However, Relational databases couldn't ever hope to achieve that, with their rigid schemas and complex development. The aforementioned rapid changes are also met by growing size, which require rapid scalability. However, scalability was one aspect that was blatantly ignored in SQL (as it was made in a time when web and internet were non-existent). To cope up with these issues, NoSQL seems like our best bet.

Why to Bridge SQL and NoSQL

"Now that we know how NoSQL differs from SQL, the question arises: Why to bridge them? Why not adopt NoSQL altogether?   "

Simply, because NoSQL doesn't have the same penetration as SQL. A huge number of companies have their entire existing architecture based on relational databases, which would be quite a headache to change. But that doesn't mean that one has to remain stuck with SQL forever. The best option in such scenarios is to bridge the existing SQL framework with a NoSQL database. The benefit? To put it simple, it will bring out "the best of both worlds".

As far the "bridging" goes, there is no one, simple way to do that. The easiest way would be to use third-party drivers like easysoft, which provides ODBC-like bridging capabilities. However, as it comes from a third-party vendor, it might have its own security and licensing issues.

An alternative approach would be to develop languages that could extend SQL functionality to NoSQL databases. One example would be the N1QL, introduced by Couchbase Server, which extends SQL to JSON.

The ways to bridge the gap between these two technologies may differ and evolve; but we can all agree that co-existence of the two is best for the progress of industry.

Please share your thoughts on this topic. If you like this posts, please share it on google by clicking on the Google +1 button.

Read more on NO SQL- NOT ONLY SQL here - WhatisNoSQL

7 Top Big Data Tools for Enterprise Developers

Big Data is now a critical technology utilized for leveraging data to enable better decision-making. Developers today have a wide variety of choice in picking a tool for their needs – open source or proprietary.

A through assessment of the existing data structure and the business requirements is essential for developers to identify the right tool. Predominant data formats, existing database types, available budget and the desired output analytics are some important factors. A few of the top tools that can be considered by developers are given below:

1.   MongoDB

This is an open source platform that that is heavily document oriented allowing for full-fledged indexing. Users can index any attribute and can also scale the data horizontally. Its a cross-platform tool that allows developers great control over final results

Read more about this at Mongo DB-Features         

2.   SAP HANA

A proprietary platform from tech giant SAP, HANA offers in-memory data storage that speeds up data processing and delivers more insightful data. It is flexible in handling all forms of data including spatial data, graph data and text data, at real time, making it a powerful though an expensive tool for holistic data analytics.

3.   Google Charts

A free open source platform from Google, this has a wide range of capabilities to handle data off a website. Primarily used for visualization, it can be plugged into a website with a simple JavaScript code. Developers can use it to create dashboards, carry out data management tasks, pull data from an external database, among other tasks

4.   Hadoop

Hadoop is the Big Data tool that everyone has been talking about. An open-source framework, it can handle, store and process massive amounts of data. Due to a distributed computing model, the data processing is fast and powerful. The tool is highly flexible and scalable, making it an ideal choice to leverage Big Data analytics for enormous data sets

Read more about this at Hadoop-Features

5.   Spark

Spark is an highly popular open-source data processing platform and is said to be the most active Apache open source project under Big Data. It is an extremely fast – 100x faster than Hadoop- and flexible platform, enabling analyses all forms of structured and unstructured data. Its advantages include use of multiple languages and accessibility to other databases. 

6.   Splice Machine

Splice Machine is a SQL-on-Hadoop database that analyses data in real time. The tool allows developers to utilize standard SQL on it, giving it flexibility and making it easier to use.

Splice Machine which is also ACID-compliant product is available on a freemium basis and has a list price annual license fee of $5,000 per node.

7.   Splunk

It’s a popular Advanced IT Search Tool used by many companies which derives information from machine data. To make it more lucid, Splunk has the ability to search, monitor and analyze through all machine generated data such as log data generated by applications, servers, and network devices across an organization.

This product further indexes structured as well as unstructured data and helps in diagnosing the problems, making it easy for administrators, business analysts and managers to detect requisite information.

Read more at Splunk-Features

TOP 6 BIG DATA TRENDS IN THE NEAR FUTURE

Big Data is a buzz word we all are familiar with now. But behind the buzz, there have been rapid developments which has changed business models and brought big data to the strategic foreground. 2016 has been a pretty eventful year for BigData and the future indicates promising. Let’s take a look at the top trends that will follow in the upcoming year:

1) CUSTOMER DIGITAL ASSISTANTS

One of the surprising trends we saw this year was the growing interests in Digital Assistants. The logic had been simple: If we could gather and process data to generate meaningful results, why do we need humans to convey them to customers?  The most devoted users are perhaps the gamers, who have fully accepted this technology in the likes of XBox One and Sony PS4. With advanced NLP and audio-recognition, mobile digital assistants like Cortana, Siri and Google Now are almost the must-haves today, and all signs indicate that digital assistants will play an even more important role in the upcoming year.

2) SIMPLER DATA ANALYSIS

Like many past years, data saw an unprecedented growth in volume and veracity. With this rate, the current data analysis techniques would soon be obsolete. However, the upcoming trend in 2017 might focus on simplifying the data analysis process, to an extent where even non-coders could easily analyze huge datasets. Giants like Microsoft and Salesforce are working upon it, while complementary tools to SQL like Spark will continue to make storage and access of data easier.

3) MACHINE LEARNING IS THE FUTURE

Not far ago, machine learning was considered purely a research field. For the benefit of all, this perception soon changed and today, machine learning has dedicated departments in numerous companies. For business purposes, the idea of machine learning is to serve as an extension to predictive analytics, thereby minimizing the work and maximizing the profits. This trend will continue to be one of the top business strategies in the future.

4) DATA-AS-A-SERVICE

Although it took a long, long time; but today, companies realize the importance of their data. This, in turn, is giving rise to an entire new business model of data-as-a-service (DaaS). With IBm's acquisition of The Weather Channel, more tech giants might realize that their data can, in fact, be converted into a profitable service.

5) THE TRANSITION OF BIG DATA to “ACTIONABLE DATA”

Big data will continue to face its existing challenges- the most prominent being the required manpower to handle the ever-increasing volume. Privacy concerns will also continue to haunt the general perception regarding the increased use of Big Data. Amidst all that is the new question: Why to worry about big data when most companies only use a fraction of it anyway? The answer to this question is giving rise to a new trend of "actionable data", data that is relevant to the business. It is entirely possible that big data may be replaced by actionable data in upcoming years.

6) INTERNET-OF-THINGS

One of the most revolutionary digital concepts of this century, IoT still fascinates masses, even if its application continues to face hurdles. But the rise and success of IoT is inevitable. With the rapid rate with which devices are becoming integral parts of our lives, IoT can provide us with unmeasured potential. While the initial cost of converting every device as a node in a vast, digital world is pretty high, it is estimated that IoT will grow by 30% in next 5 years, creating an economic value of $4-11 trillion by 2025.

If you like this post, please share it on google by clicking on the Google +1 button.

How to Create/Change/Set Databases in Hive?

As discussed in previous posts, HIVE makes it easier for developers to port SQL-based applications to Hadoop, compared with other Hadoop languages and tools.

http://dwhlaureate.blogspot.in/2013/11/apache-hive-hive-query-language.html

Hive is most suited for data warehouse applications where data is static and fast response time is not required and record-level inserts, updates, and deletes are not required

Creating a Database

The simplest syntax for creating a database in hive is shown in the following example:

Go to the Hive Shell by giving the command sudo hive and enter the command

CREATE DATABASE <DATA BASE NAME>

EXAMPLE

HIVE> CREATE DATABASE HR_STAGING;

HIVE> CREATE DATABASE IF NOT EXISTS HR_STAGING;

We can suppress the warning if the database hr_staging already exists in the hive database by using IF NOT EXISTS. The general syntax for creating the database in Hive is given below. The keyword ‘SCHEMA’ can be used instead of ‘DATABASE’ while creating database.

CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] DATABASE_NAME

  [COMMENT DATABASE_COMMENT]

  [LOCATION HDFS_PATH]

  [WITH DBPROPERTIES (PROPERTY_NAME=PROPERTY_VALUE, ...)]

The CREATE DATABASE command creates the database under HDFS at the default location: /user/hive/warehouse.

Hive creates a directory for each database. Tables in that database will be stored in sub directories of the database directory. The exception is tables in the default database, which doesn’t have its own directory.

Syntax to see the databases that already exists in hive. 

  

HIVE> SHOW DATABASES;

HR_STAGING

HIVE> CREATE DATABASE EMP_STAGING ;

HIVE> SHOW DATABASES;

HR_STAGING

EMP_STAGING

HIVE> SHOW DATABASES LIKE 'H.*';

HR_STAGING

Using a Database

The USE command sets a database as your working database, similar to changing working directories in a file system:

SYNTAX:

USE <DATABASE_NAME>

HIVE> USE HR_STAGING;

HIVE > USE DEFAULT;

Dropping Database in Hive

Syntax to drop a database:

HIVE> DROP DATABASE IF EXISTS HR_STAGING;

Hive won’t allow to drop the database if they contain tables. In such case we have to either drop the table first or append the CASCADE keyword to the command, which will cause the Hive to drop the tables in the database first.

DROP (DATABASE|SCHEMA) [IF EXISTS] DATABASE_NAME

[RESTRICT|CASCADE];

HIVE> DROP DATABASE IF EXISTS HR_STAGING CASCADE;

Alter Database in Hive

You can set key-value pairs in the DBPROPERTIES associated with a database using the ALTER DATABASE command. No other metadata about the database can be changed, including its name and directory location:

ALTER (DATABASE|SCHEMA) DATABASE_NAME

SET DBPROPERTIES (PROPERTY_NAME=PROPERTY_VALUE, ...); 

ALTER (DATABASE|SCHEMA) DATABASE_NAME

SET OWNER [USER|ROLE] USER_OR_ROLE;

HIVE> ALTER DATABASE HR_STAGING

SET DBPROPERTIES ('EDITED-BY' = 'XXXX');

If you like this post, please share it on google by clicking on the Google +1 button.