setting up hdp 2.1 with non-standard users for hadoop services (why not use a non-standard user for ambari, too)
If you find yourself needing to setup Hortonworks Data Platform (HDP) with Ambari in an environment that users and groups need to be pre-provisioned instead of simply created during the install process, then don't fret as Ambari has got you covered. This write-up piggybacks the HDP Documentation site and uses HDP 2.1.2 along with Ambari 1.5.1 as a baseline to build against. It will also build a 4-node cluster (2 worker nodes, 1 master node, and 1 node to run Knox on) all running on CentOS 6.5
As this whole post is about using non-standard users and groups, let's refer to http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-users_2x.html which lists the default user and group names. For our purposes, let's just go and and prefix all of these with "ryo
" (example: user account "mapred
" for the MapReduce2 service will now be "ryomapred
"). Also, we'll want to install, and execute, Ambari as a user other than root. Using this same general theme, we'll use "ryoambari
" for the user and group name.
To get started, follow the general flow of building a virtualized 5-node HDP 2.0 cluster (all within a mac) to set up the servers up until you get to the section called Install Cluster via Ambari. And yes, there are faster ways to get to this point such as described in https://blog.codecentric.de/en/2014/04/hadoop-cluster-automation/ using tools such as Vagrant, but there is value to walking these steps for those shoring up their linux admin skills. Back to those more lengthy instructions, start out building the First Master Node as described previously, but remember that we want to build CentOS 6.5 boxes (grabbed my ISO here). For this host name, call the VirtualBox entry 4N-HDP212-M1
to differentiate from the older 2.0 cluster. It's OK to reuse the m1.hdp2
hostname.
Work through the same network setup instructions for this first node, but pause for a second when you get to the SSH Setup section. The instructions at http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap1-5-2.html are to make sure that Ambari can make password-less SSH connections and are based on the default approach of letting Ambari run as root
. As we'll be using "ryoambari
" (mentioned above) we need to set this up when we are that user. To do that, we'll need the user in the first place. We can create the user (which creates a group with same name) and set the password to "hadoop
" for simplicity's sake.
[root@m1 ~]# useradd -m -s /bin/bash ryoambari [root@m1 ~]# passwd ryoambari Changing password for user ryoambari. New password: BAD PASSWORD: it is based on a dictionary word BAD PASSWORD: is too simple Retype new password: passwd: all authentication tokens updated successfully. [root@m1 ~]# groupadd ryoambari groupadd: group 'ryoambari' already exists [root@m1 ~]# cat /etc/group ryoambari:x:500: [root@m1 ~]# cat /etc/passwd ryoambari:x:500:500::/home/ryoambari:/bin/bash
A note is identified in http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-users_2x.html that states "All new service user accounts, and any existing user accounts used as service users, must have a UID >= 1000." Unfortunately, as described here, CentOS & RHEL begin their numbering at 500.
That's probably not a big deal for the account we are going to use for Ambari itself, but checking it out & making corrective actions on this single account will allow us later to breeze through this concern in the verification/modification step.
[root@m1 ~]# id -u ryoambari 500 [root@m1 ~]# usermod -u 1500 ryoambari [root@m1 ~]# groupmod -g 1500 ryoambari [root@m1 ~]# id -u ryoambari 1500
I simply added 1000 to the number. This will come in handy later as the UIDs for users should be greater than this value.
A note is identified in http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap1-5-2.html that (in regards to installing/running Ambari) states, "It is possible to use a non-root SSH account, if that account can execute sudo
without entering a password."
On that note there are great write-ups out there like found here, but I cheated since this is a dev-only setup (virtualized even within my Mac) and every "real" environment will have a sysadmin who knows how to do this best for their setup. I followed this thread and just did the following to grant everyone the ability to do password-less sudo
commands.
root@m1 ~]# cp /etc/sudoers ./sudoers.original [root@m1 ~]# visudo [root@m1 ~]# diff sudoers.original /etc/sudoers 108a109 > ALL ALL=(ALL) NOPASSWD: ALL [root@m1 ~]# exit logout Connection to 192.168.56.41 closed. HW10653:~ lmartin$ ssh ryoambari@m1 Warning: Permanently added 'm1' (RSA) to the list of known hosts. ryoambari@m1's password: [ryoambari@m1 ~]$ sudo vi /etc/passwd [ryoambari@m1 ~]$
Yep, that worked! I quickly backed out of that edit, but I got in without getting prompted for a password. Again, this is surely NOT the way to do this on a production environment!
Now we can circle back and work on the SSH instructions found at http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap1-5-2.html, but do them with ryoambari
instead of root
as shown below (i.e. replaying the instructions back in building a virtualized 5-node HDP 2.0 cluster (all within a mac)).
[ryoambari@m1 ~]$ sudo yum install openssh-clients [ryoambari@m1 ~]$ ssh localhost ryoambari@localhost's password: Last login: Wed May 7 09:21:26 2014 from 192.168.56.1 [ryoambari@m1 ~]$ exit logout Connection to localhost closed. [ryoambari@m1 ~]$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/ryoambari/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ryoambari/.ssh/id_rsa. Your public key has been saved in /home/ryoambari/.ssh/id_rsa.pub. The key fingerprint is: a0:6b:49:e6:27:f4:0c:2c:8b:bf:49:f8:bb:6d:07:4e ryoambari@m1.hdp2 The key's randomart image is: +--[ RSA 2048]----+ | | | | | . | | . . . | | . B S | | o BE* | |o oo*.+ | | + +oo. | | B=.. | +-----------------+ [ryoambari@m1 ~]$ cd .ssh [ryoambari@m1 .ssh]$ pwd /home/ryoambari/.ssh [ryoambari@m1 .ssh]$ ls -l total 12 -rw-------. 1 ryoambari ryoambari 1675 May 7 19:52 id_rsa -rw-r--r--. 1 ryoambari ryoambari 399 May 7 19:52 id_rsa.pub -rw-r--r--. 1 ryoambari ryoambari 391 May 7 09:29 known_hosts [ryoambari@m1 .ssh]$ cat id_rsa.pub >> authorized_keys [ryoambari@m1 .ssh]$ ls -l total 16 -rw-rw-r--. 1 ryoambari ryoambari 399 May 7 19:53 authorized_keys -rw-------. 1 ryoambari ryoambari 1675 May 7 19:52 id_rsa -rw-r--r--. 1 ryoambari ryoambari 399 May 7 19:52 id_rsa.pub -rw-r--r--. 1 ryoambari ryoambari 391 May 7 09:29 known_hosts [ryoambari@m1 .ssh]$ chmod 600 authorized_keys [ryoambari@m1 .ssh]$ ls -l total 16 -rw-------. 1 ryoambari ryoambari 399 May 7 19:53 authorized_keys -rw-------. 1 ryoambari ryoambari 1675 May 7 19:52 id_rsa -rw-r--r--. 1 ryoambari ryoambari 399 May 7 19:52 id_rsa.pub -rw-r--r--. 1 ryoambari ryoambari 391 May 7 09:29 known_hosts [ryoambari@m1 .ssh]$ cd .. [ryoambari@m1 ~]$ pwd /home/ryoambari [ryoambari@m1 ~]$ [ryoambari@m1 ~]$ ssh localhost Last login: Wed May 7 09:29:10 2014 from localhost [ryoambari@m1 ~]$ [ryoambari@m1 ~]$ cd .ssh [ryoambari@m1 .ssh]$ echo 'StrictHostKeyChecking no' >> config [ryoambari@m1 .ssh]$ cat config StrictHostKeyChecking no [ryoambari@m1 .ssh]$
Now for an important step of this deviation from a standard install. As http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-users_2x.html calls out, we are going to create users & groups ahead of time. The following commands will create the three groups that we need and, even though there is no special warning about their IDs being >= 1000, we'll do just that.
sudo groupadd ryohadoop sudo groupmod -g 2001 ryohadoop sudo groupadd ryonagios sudo groupadd ryonobody
These IDs can be verified as shown in the abbreviated output below.
[ryoambari@m1 ~]$ cat /etc/group ryoambari:x:1500: ryohadoop:x:2001: ryonagios:x:2002: ryonobody:x:2003:
Now we need to run the following commands to get all the users created and put in their appropriate groups.
sudo useradd -gryohadoop ryohdfs sudo useradd -gryohadoop ryoyarn sudo useradd -gryohadoop ryomapred sudo useradd -gryohadoop ryotez sudo useradd -gryohadoop ryohbase sudo useradd -gryohadoop ryohive sudo useradd -gryohadoop ryohcat sudo useradd -gryohadoop ryofalcon sudo useradd -gryohadoop ryostorm sudo useradd -gryohadoop ryooozie sudo useradd -gryohadoop ryopostgres sudo useradd -gryohadoop ryozookeeper sudo useradd -gryohadoop ryoambari-qa sudo useradd -gryonobody ryonobody sudo usermod -Gryohadoop ryonobody sudo useradd -gryonobody ryorrdcahed sudo usermod -Gryohadoop ryorrdcahed sudo useradd -gryonobody ryoapache sudo usermod -Gryohadoop ryoapache sudo useradd -gryonagios ryonagios sudo usermod -Gryohadoop ryonagios
Knowing that the ryoambari
UID was 1500 suggested these new users would have IDs >= 1000 as well and can easily be verified.
[ryoambari@m1 ~]$ cat /etc/passwd ryoambari:x:1500:1500::/home/ryoambari:/bin/bash ryoyarn:x:1502:2001::/home/ryoyarn:/bin/bash ryohdfs:x:1503:2001::/home/ryohdfs:/bin/bash ryomapred:x:1504:2001::/home/ryomapred:/bin/bash ryotez:x:1505:2001::/home/ryotez:/bin/bash ryohbase:x:1506:2001::/home/ryohbase:/bin/bash ryohive:x:1507:2001::/home/ryohive:/bin/bash ryohcat:x:1508:2001::/home/ryohcat:/bin/bash ryofalcon:x:1509:2001::/home/ryofalcon:/bin/bash ryostorm:x:1510:2001::/home/ryostorm:/bin/bash ryooozie:x:1511:2001::/home/ryooozie:/bin/bash ryopostgres:x:1512:2001::/home/ryopostgres:/bin/bash ryozookeeper:x:1513:2001::/home/ryozookeeper:/bin/bash ryoambari-qa:x:1514:2001::/home/ryoambari-qa:/bin/bash ryonobody:x:1515:2003::/home/ryonobody:/bin/bash ryorrdcahed:x:1516:2003::/home/ryorrdcahed:/bin/bash ryoapache:x:1517:2003::/home/ryoapache:/bin/bash ryonagios:x:1518:2002::/home/ryonagios:/bin/bash
Now you can return to the notes in building a virtualized 5-node HDP 2.0 cluster (all within a mac) ( i.e. keep on chugging through the install docs) after the SSH Setup section and work all the way through Rinse, Lather, and Repeat... as long as you take into account the change to the hosts we are trying to build and set them up as identified below. We can call the newly created VirtualBox appliance 4N-HDP212-template (with file name of 4N-HDP212-template.ova).
VirtualBox Name | OS Hostname | IP Address |
---|---|---|
4N-HDP212-K1 | k1.hdp2 | 192.168.56.31 |
4N-HDP212-M1 | m1.hdp2 | 192.168.56.41 |
4N-HDP2-W1 | w1.hdp2 | 192.168.56.51 |
4N-HDP2-W2 | w2.hdp2 | 192.168.56.52 |
When you start each VM up for the first time, log in as ryoambari
to make all the identified changes to make the box unique (prefixing everything with sudo
of course).
As I was working through the Ensure Connectivity Between Nodes section of the other post I ran into the problem below and thanks to some quick googling I found a quick fix at http://www.omnicode.com/qa.php?id=69 that did the trick. Unfortunately, I had to do this on each box.
[ryoambari@m1 ~]$ ssh k1.hdp2 Bad owner or permissions on /home/ryoambari/.ssh/config [ryoambari@m1 ~]$ cd .ssh [ryoambari@m1 .ssh]$ ls -l total 20 -rw-------. 1 ryoambari ryoambari 399 May 7 19:53 authorized_keys -rw-rw-r--. 1 ryoambari ryoambari 25 May 8 20:06 config -rw-------. 1 ryoambari ryoambari 1675 May 7 19:52 id_rsa -rw-r--r--. 1 ryoambari ryoambari 399 May 7 19:52 id_rsa.pub -rw-r--r--. 1 ryoambari ryoambari 794 May 8 20:06 known_hosts [ryoambari@m1 .ssh]$ chmod 600 * [ryoambari@m1 .ssh]$ ssh k1.hdp2 Last login: Thu May 8 20:06:02 2014 from m1.hdp2 [ryoambari@k1 ~]$ ssh m1.hdp2 Bad owner or permissions on /home/ryoambari/.ssh/config [ryoambari@k1 ~]$ cd .ssh [ryoambari@k1 .ssh]$ chmod 600 * [ryoambari@k1 .ssh]$ ssh m1.hdp2 Warning: Permanently added 'm1.hdp2,192.168.56.41' (RSA) to the list of known hosts. Last login: Thu May 8 20:05:49 2014 from localhost [ryoambari@m1 ~]$
Phew... that's done! Now we (finally) get to start installing Ambari. For the most part, just follow the Install Cluster via Ambari ramblings back in building a virtualized 5-node HDP 2.0 cluster (all within a mac). Of course, this is HDP 2.1, not 2.0, but very similar. I'll call out the major differences below.
Just to make my install a bit more interesting, I decided to go down the optional step identified at http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap1-6.html since most datacenter-based clusters will not have internet access. Thankfully, I already did all this setup on my m1.hdp2
master node where I will be installing Ambari in an earlier post; using your mac to install a virtualized hadoop cluster? (then setup a local repo on it). Feel free to just skip that if your fingers are already getting tired on this exercise.
It looks like during the "sudo ambari-server setup
" exercise discussed in http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap2-2.html I lost my chance to use the "ryopostgres
" user I created earlier by not choosing to enter the advanced database configuration. This one is OK as most enterprises will already have a centralized database they want to use anyway and the setup for that will trump this concern.
If you went down the local repo option like I did, then when you get to http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap3-2a_2x.html make sure your screen looks something like the following (i.e. use the URL you created in the prior posting).
There is a subtle callout on http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap3-3_2x.html for running Ambari as another user besides root
and the screenshot below shows what that looks like when declared to run as ryoambari
.
Unfortunately, the Confirm Hosts page showed failures for all nodes with the following messages.
ERROR 2014-05-08 22:21:18,891 NetUtil.py:56 - [Errno 1] _ssl.c:492: error:100AE081:elliptic curve routines:EC_GROUP_new_by_curve_name:unknown group ERROR 2014-05-08 22:21:18,892 NetUtil.py:58 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. INFO 2014-05-08 22:21:18,892 NetUtil.py:81 - Server at https://m1.hdp2:8440 is not reachable, sleeping for 10 seconds... ', None)
The good news is that message was referenced in http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-chap3-4_2x.html and following the information provided I found out that I needed a newer version of OpenSSL which was accomplished by running "sudo yum upgrade openssl
" on all the boxes to get past these errors.
I also found out that I had a warning from each of the four nodes that the ntpd service was not running. I thought I took care of this earlier, but either way I just followed the instructions on this back in building a virtualized 5-node HDP 2.0 cluster (all within a mac) the warnings cleared up.
Unlike the other cluster install instructions, for this setup we want all services checked on the Choose Services page and then you can take some creative liberty on the Assign Masters page. Here's a snapshot of my selections.
Here's a view of how the Assign Slaves and Clients screen looked during the installation.
The prior instructions help you clear the Hive and Oozie warnings on the Customize Services screen. The Nagios ones can be handled just as easy (i.e. just keep using "hadoop
" as the password). Also, since we only have two DataNodes it would make sense to throttle back the HDFS > Advanced > Block replication value from 3 to 2.
Critical to this blog posting is the Misc tab here on the Customize Services screen which should be setup as the following. I basically put "ryo
" in front of all except a couple of items, but be sure to read the notes below the screenshot before moving on.
Looking back on http://docs.hortonworks.com/HDPDocuments/Ambari-1.5.1.0/bk_using_Ambari_book/content/ambari-users_2x.html it seems I had some earlier misunderstandings about how this would all go. The following table identifies these and the remedial action, if any, I took to resolve each.
Item to Resolve | Action Taken |
---|---|
The Misc tab had a "Proxy group for Hive, WebHCat, Oozie and Falcon" field that I wasn't expecting | I simply left it as "users " |
The Misc tab had no place to identify the Ganglia Group of "ryonobody " that I previously created and used as the primary group for the "ryonobody " user | Knowing there is a user and a group both named "nobody " on the base OS install (and considering the bolt-on nature of Ganglia to HDP) I left the user as "nobody " |
The Misc tab had no place to identify the RRDTool's "ryorrdcahed " user that I previously created | Reading the notes again I decided (maybe realized?) this is another bolt-on service for HDP and didn't worry about the user I previously created |
The Misc tab had no place to identify the "ryoapache " user that is associated with Ganglia | Same as prior action |
The Misc tab had not place to identify the "ryopostgres " user that Ambari itself uses | No worries, but this could have been resolved during the CLI setup of Ambari as mentioned earlier |
Thankfully... the Install, Start and Test screen clear with all green bars. Now, like shown at the end of building a virtualized 5-node HDP 2.0 cluster (all within a mac), we need to make sure the ambari-server and ambari-agent services are started upon reboot. After taking care of that and restarted the virtual machines and starting all Hadoop services from Ambari, all heck broke loose. As you do in situations like this, I worked the logs and finally got some answers.
It seems the service YARN was actually kicking off as the user "yarn
" as the box started and my user "ryoyarn
" was then failing when being started from Ambari. Medium story, short, I ended up realizing I needed to remove all the users with the default names that Ambari told me I could cleanup later. Of course, I started with the "yarn
" user, but was being told I couldn't as it was logged in which was resolved with a bit of brute force (i.e. I ran "sudo su - yarn
", then "kill -9 -1
", "exit
" to return to "ryoambari
", and then kicked off the "sudo userdel yarn
" which completed. I did this on all four boxes.
I then had some permission problems with the home directories on the users I created (not sure why HDP needed to do anything with these directories) that I also solved with some brute force; "sudo chmod 777 /home/*
". I also decided I'd better go ahead and delete the other default user accounts on all boxes.
sudo userdel zookeeper sudo userdel hdfs sudo userdel mapred sudo userdel falcon sudo userdel hbase sudo userdel hive sudo userdel oozie sudo userdel sqoop sudo userdel storm
I was able to stop and start all service to green bar (although I did it service by service) after these modifications and I also restarted the OS on all the boxes to make sure I could get back to this "green" place.
This environment surely needs to be test some more to verify all is as well as it looks like it is. Also, there were a number of brute force hacks that I did because this is a dev cluster and they each warrant revisiting if you were considering going forward with something like this.
The real driver for this posting was to see that you could use different user & groups names for the service as well as run Ambari as another user than root. I believe I've shown that this is the case, but... I would not recommend this be done for a production cluster.
Final Words: Don't do this! Stick with the default user and group names to prevent as many problems (those found and those yet uncovered) as possible.