org.elasticsearch.spark.deploy.yarn.security

Class EsServiceCredentialProvider

java.lang.Object

org.elasticsearch.spark.deploy.yarn.security.EsServiceCredentialProvider

All Implemented Interfaces:

org.apache.spark.security.HadoopDelegationTokenProvider

public class EsServiceCredentialProvider
extends java.lang.Object
implements org.apache.spark.security.HadoopDelegationTokenProvider

A provider interface in Spark's Yarn library that obtains tokens for an application.

When a job is submitted to a YARN cluster, the credential providers are constructed using a service loader. Each provider is queried for if its service requires a token, and if so, Spark requests that it obtains one.

In client deployment mode, these tokens are retrieved on the driver when the YARN application is first initialized and started by the main program.

In cluster deployment mode, these tokens are retrieved initially on the driver before submitting the application master that runs the main program.

If a principal and keytab are provided to a job, a credentials file is created, and a background thread is started on the application master that will obtain new tokens when they get close to expiring. Those tokens are written to an HDFS directory which the worker nodes will regularly poll to get updated tokens. If the job is launched in client mode, the client will also receive updated tokens.

Constructor Summary

Constructors

Constructor and Description
EsServiceCredentialProvider()

Method Summary

Modifier and Type	Method and Description
boolean	delegationTokensRequired(org.apache.spark.SparkConf sparkConf, Configuration hadoopConf) Given a configuration, check to see if tokens would be required.
scala.Option<java.lang.Object>	obtainDelegationTokens(Configuration hadoopConf, org.apache.spark.SparkConf sparkConf, Credentials creds) Obtains api key tokens from Elasticsearch and stashes them in the given credentials object
java.lang.String	serviceName() Name of the service for logging purposes and for the purpose of determining if the service is disabled in the settings using the property spark.security.credentials.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EsServiceCredentialProvider

public EsServiceCredentialProvider()

Method Detail

delegationTokensRequired

public boolean delegationTokensRequired(org.apache.spark.SparkConf sparkConf,
                                        Configuration hadoopConf)

Given a configuration, check to see if tokens would be required.

Specified by:

delegationTokensRequired in interface org.apache.spark.security.HadoopDelegationTokenProvider

Parameters:

sparkConf - the current Spark configuration - used by Cloudera's CDS Spark fork (#1301)

hadoopConf - the current Hadoop configuration

Returns:

true if tokens should be gathered, false if they should not be

obtainDelegationTokens

public scala.Option<java.lang.Object> obtainDelegationTokens(Configuration hadoopConf,
                                                             org.apache.spark.SparkConf sparkConf,
                                                             Credentials creds)

Obtains api key tokens from Elasticsearch and stashes them in the given credentials object

Specified by:

obtainDelegationTokens in interface org.apache.spark.security.HadoopDelegationTokenProvider

Parameters:

hadoopConf - Hadoop configuration, picking up all Hadoop specific settings

sparkConf - All settings that exist in Spark

creds - The credentials object that will be shared between all workers

Returns:

The expiration time for the token

serviceName

public java.lang.String serviceName()

Name of the service for logging purposes and for the purpose of determining if the service is disabled in the settings using the property spark.security.credentials.[serviceName].enabled

Specified by:

serviceName in interface org.apache.spark.security.HadoopDelegationTokenProvider

Returns:

the service name this provider corresponds to