org.apache.datasketches.pig.quantiles

Class DataToItemsSketch<T>

java.lang.Object

org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

org.apache.datasketches.pig.quantiles.DataToItemsSketch<T>

Type Parameters:

T - type of item

All Implemented Interfaces:

org.apache.pig.Accumulator<org.apache.pig.data.Tuple>, org.apache.pig.Algebraic

Direct Known Subclasses:

DataToStringsSketch

public abstract class DataToItemsSketch<T>
extends org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>
implements org.apache.pig.Accumulator<org.apache.pig.data.Tuple>, org.apache.pig.Algebraic

Builds ItemsSketch from data. To assist Pig, this class implements both the Accumulator and Algebraic interfaces.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataToItemsSketch.DataToItemsSketchInitial Class used to calculate the initial pass of an Algebraic sketch operation.
static class	DataToItemsSketch.DataToItemsSketchIntermediateFinal<T> Class used to calculate the intermediate or final combiner pass of an Algebraic sketch operation.

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
DataToItemsSketch(int k, Comparator<T> comparator, org.apache.datasketches.ArrayOfItemsSerDe<T> serDe) Base constructor.

Method Summary

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple inputTuple) An Accumulator version of the standard exec() method.
void	cleanup() Cleans up the UDF state after being called using the Accumulator interface.
org.apache.pig.data.Tuple	exec(org.apache.pig.data.Tuple inputTuple) Top-level exec function.
protected T	extractValue(Object object) Override this if it takes more than a cast to convert from Pig type to type T
org.apache.pig.data.Tuple	getValue() Returns the result of the Union that has been built up by multiple calls to accumulate(org.apache.pig.data.Tuple).
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLoadCaster, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, needEndOfAllInputProcessing, progress, setEndOfAllInput, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pig.Algebraic

getFinal, getInitial, getIntermed

Constructor Detail

DataToItemsSketch

public DataToItemsSketch(int k,
                         Comparator<T> comparator,
                         org.apache.datasketches.ArrayOfItemsSerDe<T> serDe)

Base constructor.

Parameters:

k - parameter that determines the accuracy and size of the sketch. The value of 0 means the default k, whatever it is in the sketches-core library

comparator - for items of type T

serDe - an instance of ArrayOfItemsSerDe for type T

Method Detail

exec

public org.apache.pig.data.Tuple exec(org.apache.pig.data.Tuple inputTuple)
                               throws IOException

Top-level exec function. This method accepts an input Tuple containing a Bag of one or more inner Datum Tuples and returns a single Sketch as a Sketch Tuple.

If a large number of calls is anticipated, leveraging either the Algebraic or Accumulator interfaces is recommended. Pig normally handles this automatically.

Internally, this method presents the inner Datum Tuples to a new Union, which is returned as a Sketch Tuple

Types below are in the form: Java data type: Pig DataType

Input Tuple

• Tuple: TUPLE (Must contain only one field)
  • index 0: DataBag: BAG (May contain 0 or more Inner Tuples)
    • index 0: Tuple: TUPLE Datum Tuple
    • ...
    • index n-1: Tuple: TUPLE Datum Tuple

Datum Tuple

• Tuple: TUPLE (Must contain only one field)
  • index 0: T: some suitable Pig type convertible to T

Sketch Tuple

• Tuple: TUPLE (Contains exactly 1 field)
  • index 0: DataByteArray: BYTEARRAY = a serialized QuantilesSketch object.

Specified by:

exec in class org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

Parameters:

inputTuple - A tuple containing a single bag, containing Datum Tuples.

Returns:

Sketch Tuple. If inputTuple is null or empty, returns empty sketch.

Throws:

IOException - from Pig.

See Also:

"org.apache.pig.EvalFunc.exec(org.apache.pig.data.Tuple)"

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

accumulate

public void accumulate(org.apache.pig.data.Tuple inputTuple)
                throws IOException

An Accumulator version of the standard exec() method. Like exec(), accumulator is called with a bag of Datum Tuples. Unlike exec(), it doesn't serialize the sketch at the end. Instead, it can be called multiple times, each time with another bag of Datum Tuples to be input to the Union.

Specified by:

accumulate in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

Parameters:

inputTuple - A tuple containing a single bag, containing Datum Tuples.

Throws:

IOException - by Pig

See Also:

exec(org.apache.pig.data.Tuple), "org.apache.pig.Accumulator.accumulate(org.apache.pig.data.Tuple)"

getValue

public org.apache.pig.data.Tuple getValue()

Returns the result of the Union that has been built up by multiple calls to accumulate(org.apache.pig.data.Tuple).

Specified by:

getValue in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

Returns:

Sketch Tuple. (see exec(org.apache.pig.data.Tuple) for return tuple format)

See Also:

"org.apache.pig.Accumulator.getValue()"

cleanup

public void cleanup()

Cleans up the UDF state after being called using the Accumulator interface.

Specified by:

cleanup in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

See Also:

"org.apache.pig.Accumulator.cleanup()"

extractValue

protected T extractValue(Object object)

Override this if it takes more than a cast to convert from Pig type to type T

Parameters:

object - Pig object, which needs to be converted to type T

Returns:

value of type T