Security

Authentication

Currently the only supported authentication mechanism is Kerberos, which is disabled by default. For Kerberos to work, a Kerberos Key Distribution Center (KDC) is needed, which you need to provide. The secret-operator documentation states which kind of Kerberos servers are supported and how they can be configured.

1. Prepare Kerberos server

To configure HDFS and Hbase to use Kerberos you first need to collect information about your Kerberos server, e.g. hostname and port. Additionally, you need a service-user which the secret-operator uses to create principals for the HDFS services.

2. Create Kerberos SecretClass

Configure a SecretClass with a Kerberos backend as described in the secret-operator documentation. The following guide assumes you have named your SecretClass kerberos.

3. Configure HDFS to use SecretClass

The next step is to configure your HdfsCluster to use the newly created SecretClass. Please follow the HDFS security guide to set up and test this. Please watch out to use the SecretClass named kerberos.

4. Configure HBase to use SecretClass

The last step is to configure the same SecretClass for HBase, which is done similar to HDFS.

HDFS and HBase need to use the same SecretClass (or at least use the same underlying Kerberos server). 
spec:
  clusterConfig:
    authentication:
      tlsSecretClass: tls # Optional, defaults to "tls"
      kerberos:
        secretClass: kerberos # Put your SecretClass name in here

The kerberos.secretClass is used to give HBase the possibility to request keytabs from the secret-operator.

The tlsSecretClass is needed to request TLS certificates, used e.g. for the Web UIs.

5. Verify that Kerberos authentication is required

Shell into the hbase-master-default-0 Pod and execute the following commands:

  1. kdestroy (just in case you run kinit in the Pod already in the past)

  2. echo 'list;' | bin/hbase shell

The last command should fail with the error message ERROR: Found no valid authentication method from options. You can also check the RestServer by calling curl -v --insecure https://hbase-restserver-default:8081, which should return HTTP ERROR 401 Authentication required.

6. Access HBase

In case you want to access your HBase it is recommended to start up a client Pod that connects to HBase, rather than shelling into the master. We have an integration test for this exact purpose, where you can see how to connect and get a valid keytab.

Authorization

This is an experimental feature available from HBase version 2.6.0 upwards.

Authorization using the Open Policy Agent (OPA) is supported. A custom access controller coprocessor delegates authorization decisions to OPA. For more information on the OPA authorization mechanism and how to use it within the Stackable Platform, see the OPA concepts documentation.

Authentication with Kerberos is required when using authorization.

Authorization configuration is split in two parts: 1. The OPA authorizer configuration. 2. The Rego rules that define the policy decisions.

OPA authorizer configuration

The following snippet shows how to enable authorization for an HBase cluster using OPA.

spec:
  clusterConfig:
    authorization:
      opa:
        configMapName: opa (1)
        package: hbase (2)
1 The name of the discovery ConfigMap generated by an OPA stacklet. This is by convention the name of the OPA cluster.
2 The package is the name of the rego package that contains the rules.

The operator automatically configures the HBase coprocessor with the following properties:

  • hbase.coprocessor.master.classes, hbase.coprocessor.region.classes and hbase.coprocessor.regionserver.classes are set to tech.stackable.hbase.OpenPolicyAgentAccessController

  • hbase.security.authorization.opa.policy.url is read from the discovery ConfigMap of the OPA cluster

  • hbase.security.authorization.opa.policy.dryrun : false

  • hbase.security.authorization.opa.policy.cache.active : true

  • hbase.security.authorization.opa.policy.cache.seconds : 300 (5 minutes)

  • hbase.security.authorization.opa.policy.cache.size : 1000

You can override these properties for each HBase role using the <role>.configOverrides.hbase-site.xml property.

Rego rules

Currently, rego rules are deployed as ConfigMaps that are automatically discovered by the OPA cluster. This guide assumes your OPA cluster is called opa.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: hbase-regorules
  labels:
    opa.stackable.tech/bundle: "true"
data:
  hdfs.rego: |
    package hbase

    import rego.v1

    default allow = true

This rego rule is intended for demonstration purposes and allows every operation. For a production setup you will probably need to have something much more granular. We provide a more representative rego rule in our integration tests and in the aforementioned coprocessor repository. Details can be found below in the fine-granular rego rules section.

How it works

This implementation takes an approach to HBase authorization that is fundamentally different from the default one (which relies on HDFS group mappings) and departs significantly from traditional Hadoop patterns and POSIX-style permissions.

The current rego rules ignore file ownership and permissions, and ACLs are persisted neither in ZooKeeper nor internal HBase tables. Keeping this state in HDFS/HBase clashes with the infrastructure-as-code approach (IaC).

Instead, HBase will send a request detailing who (e.g. alice/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL) is trying to execute what type of action (e.g. READ, WRITE, CREATE or ADMIN) on what namespace or table (e.g. developers:, developers/table1) to OPA. OPA then makes a decision whether this action is allowed or not.

Instead of using the HBase shell to grant or revoke rights to users, you can create rules for OPA using the Rego language to define what a specific user is allowed or not allowed to do to.

Fine-granular rego rules

Access is granted by looking at three bits of information that must be supplied for every rego-rule callout:

  • the identity of the user

  • the resource requested by the user

  • the action which the user wants to perform on the resource

Each operation has an implicit action-level attribute e.g. CREATE requires at least read-write permissions. This action attribute is then checked against the permissions assigned to the user by an ACL and the operation is permitted if this check is fulfilled.

The basic structure of this rego rule is shown below.

Rego rule outline
package hbase

import rego.v1

# Turn off access by default.
default allow := false
default matches_identity(identity) := false

# Table is null if the request is for namespace permissions, but as parameters cannot be
# undefined, we have to set it to something specific:
checked_table_name := input.table.qualifierAsString if {input.table.qualifierAsString}
checked_table_name := "__undefined__" if {not input.table.qualifierAsString}

# Check access in order of increasing specificity (i.e. identity first).
# Deny access as "early" as possible.
allow if {
    some acl in acls
    matches_identity(acl.identity)
    matches_resource(input.namespace, checked_table_name, acl.resource)
    action_sufficient_for_operation(acl.action, input.action)
}

# Identity checks based on e.g.
# - explicit matches on the fully-qualified userName
# - regex matches
# - the group membership (simple- or regex-matches on fully-qualified userName)
matches_identity(identity) if {
    ...
}

# Resource checks on e.g.
# - wildcard access (for e.g. admins)
# - explicit namespace- or table-mentions
matches_resource(file, resource) if {
    ...
}

# Check the operation and its implicit action against an ACL
action_sufficient_for_operation(action, operation) if {
    action_hierarchy[action][_] == action_for_operation[operation]
}

action_hierarchy := {
    "full": ["full", "rw", "ro"],
    "rw": ["rw", "ro"],
    "ro": ["ro"],
}

action_for_operation := {
    "ADMIN": "full",
    "CREATE": "full",
    "WRITE": "rw",
    "READ": "ro",
}

# Maps users to groups
groups_for_user := {
    "fully-qualified userName": ["ACL group"],
    ...
}

acls := [
    {
        # Identity: either "group:groupName" or "user:userName"
        "identity": "group:admins",
        "action": "full",
        "resource": ":",
    },
    {
        "identity": "group:developers",
        "action": "full",
        "resource": "developers:",
    },
    ...
]

There is a complete file shown below, together with a list of test cases.

<details> <summary>Rego rules</summary>

package hbase

import rego.v1

default allow := false
default matches_identity(identity) := false

# table is null if the request is for namespace permissions, but as parameters cannot be
# undefined, we have to set it to something specific:
checked_table_name := input.table.qualifierAsString if {input.table.qualifierAsString}
checked_table_name := "__undefined__" if {not input.table.qualifierAsString}

allow if {
    some acl in acls
    matches_identity(acl.identity)
    matches_resource(input.namespace, checked_table_name, acl.resource)
    action_sufficient_for_operation(acl.action, input.action)
}

# Identity mentions the (long) userName explicitly
matches_identity(identity) if {
    identity in {
        concat("", ["user:", input.callerUgi.userName])
    }
}

# Identity regex matches the (long) userName
matches_identity(identity) if {
    match_entire(identity, concat("", ["userRegex:", input.callerUgi.userName]))
}

# Identity mentions group the user is part of (by looking up using the (long) userName)
matches_identity(identity) if {
    some group in groups_for_user[input.callerUgi.userName]
    identity == concat("", ["group:", group])
}

# Allow all resources
matches_resource(namespace, table, resource) if {
    resource == "hbase:"
}

# Allow all namespaces
matches_resource(namespace, table, resource) if {
    resource == "hbase:namespace:"
}

# Resource mentions the namespace explicitly
matches_resource(namespace, table, resource) if {
    resource == concat(":", ["hbase:namespace", namespace])
}

# Resource mentions the namespaced table explicitly
matches_resource(namespace, table, resource) if {
    resource == concat("", ["hbase:table:", namespace, "/", table])
}

match_entire(pattern, value) if {
	# Add the anchors ^ and $
	pattern_with_anchors := concat("", ["^", pattern, "$"])

	regex.match(pattern_with_anchors, value)
}

action_sufficient_for_operation(action, operation) if {
    action_hierarchy[action][_] == action_for_operation[operation]
}

action_hierarchy := {
    "full": ["full", "rw", "ro"],
    "rw": ["rw", "ro"],
    "ro": ["ro"],
}

action_for_operation := {
    "ADMIN": "full",
    "CREATE": "full",
    "WRITE": "rw",
    "READ": "ro",
}

groups_for_user := {
    "hbase/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": ["admins"],
    "testuser/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": ["admins"],
    "admin/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": ["admins"],
    "alice/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": ["developers"],
    "readonlyuser1/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": [],
    "readonlyuser2/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": [],
    "bob/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL": []
}

acls := [
    {
        "identity": "group:admins",
        "action": "full",
        "resource": "hbase:",
    },
    {
        "identity": "group:developers",
        "action": "full",
        "resource": "hbase:namespace:developers",
    },
    {
        "identity": "user:alice/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
        "action": "rw",
        "resource": "hbase:table:developers/table2",
    },
    {
        "identity": "user:bob/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
        "action": "rw",
        "resource": "hbase:table:developers/table1",
    },
    {
        "identity": "user:bob/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
        "action": "rw",
        "resource": "hbase:table:public/table3",
    },
    {
        "identity": "user:readonlyuser1/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
        "action": "ro",
        "resource": "hbase:table:public/test",
    },
    {
        "identity": "user:readonlyuser2/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
        "action": "ro",
        "resource": "hbase:namespace:",
    },
]

</details>

<details> <summary>Rego tests</summary>

package hbase

import rego.v1

test_permission_admin if {
    allow with input as {
    "callerUgi" : {
      "userName" : "admin/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "hbase",
      "qualifierAsString" : "meta",
    },
    "namespace" : "hbase",
    "action" : "WRITE"
    }
}

test_namespace_admin if {
    allow with input as {
    "callerUgi" : {
      "realUser" : null,
      "userName" : "admin/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "shortUserName" : "admin",
      "primaryGroup" : null,
      "groups" : [ ],
      "authenticationMethod" : "KERBEROS",
      "realAuthenticationMethod" : "KERBEROS"
    },
    "table" : null,
    "namespace" : "developers",
    "action" : "ADMIN"
    }
}

test_permission_developers if {
    allow with input as {
    "callerUgi" : {
      "userName" : "alice/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "developers",
      "qualifierAsString" : "table1",
    },
    "namespace" : "developers",
    "action" : "WRITE"
    }
}

test_permission_alice if {
    allow with input as {
    "callerUgi" : {
      "userName" : "alice/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "developers",
      "qualifierAsString" : "table2",
    },
    "namespace" : "developers",
    "action" : "WRITE"
    }
}

test_no_permission_bob if {
    not allow with input as {
    "callerUgi" : {
      "userName" : "bob/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "developers",
      "qualifierAsString" : "table2",
    },
    "namespace" : "developers",
    "action" : "WRITE"
    }
}

test_permission_bob1 if {
    allow with input as {
    "callerUgi" : {
      "userName" : "bob/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "public",
      "qualifierAsString" : "table3",
    },
    "namespace" : "public",
    "action" : "WRITE"
    }
}

test_permission_bob2 if {
    allow with input as {
    "callerUgi" : {
      "userName" : "bob/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "developers",
      "qualifierAsString" : "table1",
    },
    "namespace" : "developers",
    "action" : "WRITE"
    }
}

test_permission_hbase if {
    allow with input as {
    "callerUgi" : {
      "realUser" : null,
      "userName" : "hbase/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "shortUserName" : "hbase",
      "primaryGroup" : null,
      "groups" : [ ],
      "authenticationMethod" : "KERBEROS",
      "realAuthenticationMethod" : "KERBEROS"
    },
    "table" : {
      "name" : "aGJhc2U6bWV0YQ==",
      "nameAsString" : "hbase:meta",
      "namespace" : "aGJhc2U=",
      "namespaceAsString" : "hbase",
      "qualifier" : "bWV0YQ==",
      "qualifierAsString" : "meta",
      "nameWithNamespaceInclAsString" : "hbase:meta"
    },
    "namespace" : "hbase",
    "action" : "WRITE"
    }
}

test_permission_testuser if {
    allow with input as {
    "callerUgi" : {
      "realUser" : null,
      "userName" : "testuser/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "shortUserName" : "testuser",
      "primaryGroup" : null,
      "groups" : [ ],
      "authenticationMethod" : "KERBEROS",
      "realAuthenticationMethod" : "KERBEROS"
    },
    "table" : {
      "name" : "dGVzdA==",
      "nameAsString" : "test",
      "namespace" : "ZGVmYXVsdA==",
      "namespaceAsString" : "default",
      "qualifier" : "dGVzdA==",
      "qualifierAsString" : "test",
      "nameWithNamespaceInclAsString" : "default:test"
    },
    "namespace" : "default",
    "action" : "WRITE"
    }
}

test_permission_readonlyuser1 if {
    allow with input as {
    "callerUgi" : {
      "realUser" : null,
      "userName" : "readonlyuser1/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "shortUserName" : "readonlyuser",
      "primaryGroup" : null,
      "groups" : [ ],
      "authenticationMethod" : "KERBEROS",
      "realAuthenticationMethod" : "KERBEROS"
    },
    "table" : {
      "name" : "cHVibGljOnRlc3Q=",
      "nameAsString" : "public:test",
      "namespace" : "cHVibGlj",
      "namespaceAsString" : "public",
      "qualifier" : "dGVzdA==",
      "qualifierAsString" : "test",
      "nameWithNamespaceInclAsString" : "public:test"
    },
    "namespace" : "public",
    "action" : "READ"
    }
}

test_permission_readonlyuser2 if {
    allow with input as {
    "callerUgi" : {
      "realUser" : null,
      "userName" : "readonlyuser2/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "shortUserName" : "readonlyuser",
      "primaryGroup" : null,
      "groups" : [ ],
      "authenticationMethod" : "KERBEROS",
      "realAuthenticationMethod" : "KERBEROS"
    },
    "table" : {
      "name" : "cHVibGljOnRlc3Q=",
      "nameAsString" : "public:test",
      "namespace" : "cHVibGlj",
      "namespaceAsString" : "public",
      "qualifier" : "dGVzdA==",
      "qualifierAsString" : "test",
      "nameWithNamespaceInclAsString" : "public:test"
    },
    "namespace" : "public",
    "action" : "READ"
    }
}

</details>

If the Opa CLI tool has been installed, the tests can be executed by calling the following from the root folder:

opa test ./docs/modules/hbase/examples/rego/ -bv

e.g.
17:44 $ opa test ./docs/modules/hbase/examples/rego/ -bv
docs/modules/hbase/examples/rego/hbase_test.rego:
data.hbase.test_permission_admin: PASS (2.055153ms)
data.hbase.test_namespace_admin: PASS (528.212µs)
data.hbase.test_permission_developers: PASS (896.004µs)
data.hbase.test_permission_alice: PASS (1.56663ms)
data.hbase.test_no_permission_bob: PASS (787.445µs)
data.hbase.test_permission_bob1: PASS (1.485691ms)
data.hbase.test_permission_bob2: PASS (1.459236ms)
data.hbase.test_permission_hbase: PASS (1.524541ms)
data.hbase.test_permission_testuser: PASS (1.036205ms)
data.hbase.test_permission_readonlyuser1: PASS (780.05µs)
data.hbase.test_permission_readonlyuser2: PASS (1.37649ms)

Take the test case below as an example:

test_permission_developers if {
    allow with input as {
    "callerUgi" : {
      "userName" : "alice/test-hbase-permissions.default.svc.cluster.local@CLUSTER.LOCAL",
      "primaryGroup" : "admin",
    },
    "table" : {
      "namespaceAsString" : "developers",
      "qualifierAsString" : "table1",
    },
    "namespace" : "developers",
    "action" : "WRITE"
    }
}

This test passes through the following steps:

1. Does the user or group exist in the ACL?

Yes, a match is found on userName in the mapping groups_for_user.

2. Does this user/group have permission to fulfill the specified operation on the given path?

Yes, as this ACL item

{
    "identity": "group:developers",
    "action": "full",
    "resource": "developers:",
},

matches the resource on

# Resource mentions the namespace explicitly
matches_resource(namespace, table, resource) if {
    resource == concat("", [namespace, ":"])
}

and the action permission required for the action WRITE (rw) is a subset of the ACL grant (full).

The various checks for matches_identity and matches_resource are generic, given that the internal list of HBase actions is comprehensive and the input structure is an internal implementation. This means that only the ACL needs to be adapted to specific customer needs.

Wire encryption

In case Kerberos is enabled, Privacy mode is used for best security. Wire encryption without Kerberos as well as other wire encryption modes are not supported.