org.elasticsearch.spark.sql.streaming

Class EsSinkMetadataLog

java.lang.Object

org.apache.spark.sql.execution.streaming.HDFSMetadataLog<T[]>

org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

org.elasticsearch.spark.sql.streaming.EsSinkMetadataLog

All Implemented Interfaces:

org.apache.spark.internal.Logging, org.apache.spark.sql.execution.streaming.MetadataLog<EsSinkStatus[]>

public class EsSinkMetadataLog
extends org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

Provides logic for managing batch ID committing as well as cleaning up state between batch executions for the EsSparkSqlStreamingSink

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
EsSinkMetadataLog(Settings settings, org.apache.spark.sql.SparkSession sparkSession, java.lang.String path)

Method Summary

Modifier and Type	Method and Description
scala.collection.immutable.Seq<EsSinkStatus>	compactLogs(scala.collection.immutable.Seq<EsSinkStatus> logs) Compatibility requirement: Spark 3.0 and below used this method instead of shouldRetain.
protected int	defaultCompactInterval()
protected long	fileCleanupDelayMs()
protected boolean	isDeletingExpiredLog()
boolean	shouldRetain(EsSinkStatus log, long currentTime)
static int	VERSION_NUMBER()

Methods inherited from class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog

add, allFiles, batchIdToPath, COMPACT_FILE_SUFFIX, COMPACT_LATENCY_WARN_THRESHOLD_MS, compactInterval, deriveCompactInterval, deserialize, filterInBatch, foreachInBatch, getAllValidBatches, getBatchIdFromFileName, getValidBatchesBeforeCompactionBatch, isBatchFile, isCompactionBatch, minBatchesToRetain, nextCompactionBatchId, pathToBatchId, purge, serialize

Methods inherited from class org.apache.spark.sql.execution.streaming.HDFSMetadataLog

addNewBatchByStream, applyFnToBatchByStream, applyFnToBatchByStream$default$2, batchCache, batchFilesFilter, fileManager, get, get, getLatest, getLatestBatchId, getOrderedBatchFiles, getPrevBatchFromStorage, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, listBatches, listBatchesOnDisk, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, metadataCacheEnabled, metadataPath, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, purgeAfter, validateVersion, verifyBatchIds, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initLock, uninitialize

Constructor Detail

EsSinkMetadataLog

public EsSinkMetadataLog(Settings settings,
                         org.apache.spark.sql.SparkSession sparkSession,
                         java.lang.String path)

Method Detail

VERSION_NUMBER

public static int VERSION_NUMBER()

fileCleanupDelayMs

protected long fileCleanupDelayMs()

Specified by:

fileCleanupDelayMs in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

isDeletingExpiredLog

protected boolean isDeletingExpiredLog()

Specified by:

isDeletingExpiredLog in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

defaultCompactInterval

protected int defaultCompactInterval()

Specified by:

defaultCompactInterval in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>

compactLogs

public scala.collection.immutable.Seq<EsSinkStatus> compactLogs(scala.collection.immutable.Seq<EsSinkStatus> logs)

Compatibility requirement: Spark 3.0 and below used this method instead of shouldRetain.

This method was renamed in 3.1 but can still be used in 3.0 as long as it is present at runtime.

Parameters:

logs - (undocumented)

Returns:

(undocumented)

shouldRetain

public boolean shouldRetain(EsSinkStatus log,
                            long currentTime)

Overrides:

shouldRetain in class org.apache.spark.sql.execution.streaming.CompactibleFileStreamLog<EsSinkStatus>