org.apache.datasketches.pig.tuple

Class DataToSketch<U,S extends org.apache.datasketches.tuple.UpdatableSummary<U>>

java.lang.Object

org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

org.apache.datasketches.pig.tuple.DataToSketch<U,S>

Type Parameters:

U - Update type

S - Summary type

All Implemented Interfaces:

org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

Direct Known Subclasses:

DataToDoubleSummarySketch

public abstract class DataToSketch<U,S extends org.apache.datasketches.tuple.UpdatableSummary<U>>
extends org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>
implements org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

This is a generic implementation to be specialized in concrete UDFs.

Note Strings as values are normally typed as DataType.CHARARRAY, which will be encoded as UTF-8 prior to being submitted to the sketch. If the user requires a different encoding for cross-platform compatibility, it is recommended that these values be encoded prior to being submitted in a DataBag and then typed as a DataType.BYTEARRAY.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
DataToSketch(int sketchSize, float samplingProbability, org.apache.datasketches.tuple.SummaryFactory<S> summaryFactory) Constructs a function given a sketch size, sampling probability and summary factory
DataToSketch(int sketchSize, org.apache.datasketches.tuple.SummaryFactory<S> summaryFactory) Constructs a function given a sketch size, summary factory and default sampling probability of 1.
DataToSketch(org.apache.datasketches.tuple.SummaryFactory<S> summaryFactory) Constructs a function given a summary factory, default sketch size and default sampling probability of 1.

Method Summary

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple inputTuple)
void	cleanup()
org.apache.pig.data.Tuple	exec(org.apache.pig.data.Tuple inputTuple)
org.apache.pig.data.Tuple	getValue()

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLoadCaster, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, needEndOfAllInputProcessing, outputSchema, progress, setEndOfAllInput, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataToSketch

public DataToSketch(org.apache.datasketches.tuple.SummaryFactory<S> summaryFactory)

Constructs a function given a summary factory, default sketch size and default sampling probability of 1.

Parameters:

summaryFactory - an instance of SummaryFactory

DataToSketch

public DataToSketch(int sketchSize,
                    org.apache.datasketches.tuple.SummaryFactory<S> summaryFactory)

Constructs a function given a sketch size, summary factory and default sampling probability of 1.

Parameters:

sketchSize - parameter controlling the size of the sketch and the accuracy. It represents nominal number of entries in the sketch. Forced to the nearest power of 2 greater than given value.

summaryFactory - an instance of SummaryFactory

DataToSketch

public DataToSketch(int sketchSize,
                    float samplingProbability,
                    org.apache.datasketches.tuple.SummaryFactory<S> summaryFactory)

Constructs a function given a sketch size, sampling probability and summary factory

Parameters:

sketchSize - parameter controlling the size of the sketch and the accuracy. It represents nominal number of entries in the sketch. Forced to the nearest power of 2 greater than given value.

samplingProbability - parameter from 0 to 1 inclusive

summaryFactory - an instance of SummaryFactory

Method Detail

accumulate

public void accumulate(org.apache.pig.data.Tuple inputTuple)
                throws IOException

Specified by:

accumulate in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

Throws:

IOException

cleanup

public void cleanup()

Specified by:

cleanup in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

getValue

public org.apache.pig.data.Tuple getValue()

Specified by:

getValue in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

exec

public org.apache.pig.data.Tuple exec(org.apache.pig.data.Tuple inputTuple)
                               throws IOException

Specified by:

exec in class org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

Throws:

IOException