My 2014 prediction for IT industry

Hope 2014 will come with more positive wave and the world will regain its economic standing. So I believe there will also be a huge change in IT industry.

Change in CIO’s reporting structure

Time has come to change the reporting structure in corporate IT. Due to economic downturn few years back it was obvious to concentrate more on budget, cost and other financial issues and it was justified to report to CFO due to those facts. And the result of that change was – CIO became bureaucratic, less innovative, slow and a flat organization. This ultimately helps MSPs (Managed Service Provider) to grow like anything!!

However, the mammoths are shaking their body again and we have seen a positive sign of overcoming the business situation in 2013 which will continue in 2014 also and there will be a confidence of doing more and more investment. In this current world there is no alternative of innovation to beat the competitors and Information Technology has the inbuilt characteristics of being innovative.

Business will start demanding more from CIO to support customer needs which will create an internal conflict between CFO and CEO since CFO focuses more on internal processes. Finally CEO will win and CIO will start reporting to CEO. Who knows – may be with a new title – CPO (Chief Productivity Officer) 😉

The death of Corporate IT – we mourn!!

YES!! That will happen gradually. Let me ask you a question – Do you have electrician in your organization as a full-time employee to look after your Light, Air condition and other electronic stuff? –Definitely NOT!! Than why do you need people in your payroll to look after your Desktop/laptop issue, network and other IT equipment?

The size of Corporate IT is 60% less than that of 5 years back. Now we have so many Managed Service Providers who are really specialist on those day to day IT tasks. Moreover, business now wants to focus on their core activities and they are transferring their risk/administrative tasks to third party, so that they can concentrate more on their core businesses. There will only be few positions available for IT governance, IT Security and Vendor management.

The rise of Vertical IT

Well, business will not dare to say “BYE BYE IT”!!

As I have mentioned earlier Information Technology is the part and parcel of modern business, there will be a group of people who have core business knowledge and who can automate the business process to get best outcome. Here comes Vertical IT where technology and business knowledge blend together.

The change in BIG DATA concept

Can you remember the year 2000? People were crazy about WAP technology. Everybody started converting their web sites to be WAP enabled. By the way, do you see this word in the recent days? I am sure “NO”. But does it mean WAP is vanished? Definitely not – it improves and comes with different name. Now most of the web site can render same site as a desktop or mobile version.
I believe BIG DATA is also passing the same transition period. People yet to realize the usability and power of this concept and it will gradually evolve, maybe with a different name. OLD WINE IN A NEW BOTTLE!!

I wish you all a happy new year!!

Configuring Hadoop cluster with multiple hosts in Cloudera CDH4

I have started loving Cloudera 🙂 . So far it is the best administrative tool I have ever used. The deployment tool (Cloudera manager) is robust and user friendly and I believe it has some inbuilt AI to troubleshoot your hadoop cluster more easily.
Thanks to the Cloudera team for their continuous effort for such robust system development.

After installing my first CDH4 hadoop distribution in a single node, I decided to add a new host to the cluster. Cloudera manager is a very smart and easy to understand web tool which you can use to deploy new host in your hadoop environment.

Current configuration of my hadoop cluster

Before starting the new host configuration check whether you can ping that targeted host by hostname otherwise your installation will fail.

To mitigate this issue you can follow the below steps

1. From terminal window enter the below command
Sudoedit /etc/hosts
2. An editor will open. Enter your target hostname and IP as below and save

Hadoop cluster configuration

1. Go to the computer in which you have install CDH4
2. Open browser and enter the url – http://localhost:7180 to open Cloudera manager
3. I am still using the default user name and password which CDH4 will configure while installing the Cloudera manager (User Name: admin, Password: admin)

4. Click on Hosts menu from the top bar of the Cloudera Manager portal

5. Click on Add New Hosts to Cluster

6. Click on continue
7. Enter the IP address of the new host to which you want to install CDH4. You need to install open SSH (https://help.ubuntu.com/10.04/serverguide/openssh-server.html) otherwise the system will not be able to find the targeted host. Click on Search button

8. Select the hostname from the list and click on Continue

9. Select Latest Release of CDH4 and leave the other option as it is and click on continue

10. Enter the root password of the targeted host and click on Continue

11. The system will start installing on that targeted host

12. Based on your network speed it will take some time to complete the installation since it will download the required packages from web. You can see the current status by clicking on Details link


13. After completing the installation you can now see multiple hosts in your hadoop cluster

Add new instance in a service

1. Go to Services and click on the service to which you want to add instance that you have configured newly

2. Click on Instances link and click on Add

3. Select the role you want to assign on the targeted host and click on Continue

4. The role will be added for that service as below

5. Start the Service by selecting that and clicking on Start from Action menu

How to copy file to Hadoop file system

Due to some security restriction you will see access denied error message while transferring file to HDFS from your local file system. You can follow the below steps to copy file from local system to Hadoop file system.

At first check the current directory status
1. Open terminal
2. Enter the below command

hadoop fs -ls /

To create a directory under user folder enter the below command (here root is the directory name)

sudo -u hdfs hadoop fs -mkdir /user/root

After creating the directory, assign permission to that directory so that root user can copy data to hadoop file system.
sudo -u hdfs hadoop fs -chown root:root /user/root

If you use the normal hadoop mkdir command without using sudo command you may see the below permission denied error.

To copy file enter the below command (assuming that you have a file in your Documents folder name Test.txt and you want to copy it in hadoop user/root folder)

hadoop fs -copyFromLocal Documents/Test.txt hdfs://localhost/user/root/Test.txt

Here you may see the connection refused error if you are using Cloudera hadoop version (CDH4)

In this case you should use host name instead of localhost as below (here smr01 is my hostname)

hadoop fs -copyFromLocal Documents/Test.txt hdfs://smr01/user/root/Test.txt

Now you can see that your file has been copied in HDFS