...
As a result of Hortonworks’ recent acquisition of XA Secure, the addition of a unified console for authorization across the ecosystem offerings has become available. This mature product The maturity of this product, which is actively being spun into the new incubating project called Apache Argus which , should greatly help its eventual promotion to a top-level Apache project.
What are people doing in this space? Those already using HBase are leveraging its features as appropriate to their projects, but HDFS ACLs and Hive's dfjdjfkdjfkjfdkfdjhe additional ACL abilities can become useful as Mercy’s applications and expertise mature, but the recommendation is to first address data access controls via the base POSIX permission model as this will the first line of defense for CLI access to the file system and with ecosystem tools like Pig. Attention needs to be paid to this level of authorization especially considering the approach of limiting access on data, not the tools. ATZ-NG are still gaining adoption due to their relatively recent introductions.
Where do I go for more info? http://hbase.apache.org/book/hbase.accesscontrol.configuration.html, https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization, http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.5/bk_system-admin-guide/content/ch_acls-on-hdfs.html and http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_HDPSecure_Admin/content/ch_XA-overview.html
Activity Logging
HDFS has a built in ability to log access requests to the filesystem. This provides a low-level snapshot of events that occur and who did them. These are human-readable, but not necessarily reader-friendly. They are detailed logs that can themselves be used as a basis for reporting and/or ad-hoc querying with Hadoop frameworks and/or other 3rd party tools.
Hive also has an ability to log metastore API invocations. Additionally, HBase offers its own audit log. To pull this all together into a single pane of glass , Hortonworks offers you can leverage XA Secure / Apache Argus. This software layer pulls this dispersed audit information into a cohesive user experience. The audit data can be further broken down into access audit, policy audit and agent audit data, giving granular visibility to auditors on users’ access as well as administrative actions within this security portal.
This feature set, coupled with the single pane of glass for authorization rights, seems to be a solid fit for Mercy who, by nature of the data being stored, will require this level of administrative controls. Additional investigations are warranted to ensure XA Secure will meet the end goals of the auditing requirements that are present and additional hardware will need to be secured to run this web tier service.
Mercy should ensure that appropriate levels of auditing information is being produced and maintained to support any requirements or desired reporting opportunities
Perimeter Security
Establishing What are people doing in this space? XA Secure / Apache Argus is early in its adoption curve, but is a feature-rich application so I do expect quick adoption of it. While the raw data is there, it seems that few organizations are currently rolling the disparate activity logs into a combined view for audit reporting.
Where do I go for more info? http://books.google.com/books?id=drbI_aro20oC&pg=PA346&lpg=PA346&dq=hdfs+audit+log&source=bl&ots=t_wnyhn0i4&sig=PEbiD5LdLdkUP0jnjhtUoOoBDMM&hl=en&sa=X&ei=YwsJVKeGDZSBygT5x4D4Cg&sqi=2&ved=0CEIQ6AEwAg#v=onepage&q=hdfs%20audit%20log&f=false and http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_HDPSecure_Admin/content/ch_XA-audit.html
Perimeter Security
As mentioned at the beginning of this post, establishing perimeter security has been a foundational Hadoop security approach. This is simply viewed as having limited access into the HDP Hadoop cluster itself and utilizing an “edge” node (aka gateway) as a host users and systems can utilize directly and then have all other actions fired from this gateway into the cluster itself.
The Knox Gateway is a recent addition to the HDP stack and Apache Knox gateway provides a software layer intended to perform this perimeter security function. Not all operations are fully supported yet with Knox to have it completely replace the need for the traditional edge node (ex: HS2 access only supported for JDBC; ODBC functionality still lacking).
Mercy could benefit from additional investigations into Knox at this time, but would need to provision additional hosts and wall off the Knox Gateway web server farm with firewalls similar to the network architecture diagram below.
KNOX PICTURE
The traditional edge node also often is satisfying the ingestion node requirements that are in place in many models that need a landing zone for data before being persisted into HDFS. Mercy’s current investment in in traditional edge nodes is a meaningful one, but the project's roadmap addresses missing functionality.
What are people doing in this space? Early adopters have already deployed Knox, but the majority of clusters still rely heavily on traditional edge nodes.
Where do I go for more info? http://www.dummies.com/how-to/content/edge-nodes-in-hadoop-clusters.html and http://knox.apache.org/
Data Encryption
Data encryption can be broken into two primary scenarios; encryption in-transit and encryption at-rest. Wire encryption options exist in Hadoop to aid with the in-transit needs that might be present. HDP has many options available to protect data as it moves through Hadoop over RPC, HTTP, Data Transfer Protocol (DTP), and JDBC.
...