org.apache.datasketches.pig.kll

Class DataToSketch

java.lang.Object

org.apache.pig.EvalFunc<org.apache.pig.data.DataByteArray>

org.apache.datasketches.pig.kll.DataToSketch

All Implemented Interfaces:

org.apache.pig.Accumulator<org.apache.pig.data.DataByteArray>, org.apache.pig.Algebraic

public class DataToSketch
extends org.apache.pig.EvalFunc<org.apache.pig.data.DataByteArray>
implements org.apache.pig.Accumulator<org.apache.pig.data.DataByteArray>, org.apache.pig.Algebraic

This UDF is to build sketches from data. This class implements both the Accumulator and Algebraic interfaces for performance optimization.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataToSketch.Final Class used to calculate the final pass of an Algebraic sketch operation.
static class	DataToSketch.Initial Class used to calculate the initial pass of an Algebraic sketch operation.
static class	DataToSketch.Intermediate Class used to calculate the intermediate pass of an Algebraic sketch operation.

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
DataToSketch() Default constructor.
DataToSketch(String kStr) String constructor.

Method Summary

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple inputTuple) An Accumulator version of the standard exec() method.
void	cleanup() Cleans up the UDF state after being called using the Accumulator interface.
org.apache.pig.data.DataByteArray	exec(org.apache.pig.data.Tuple inputTuple) Top-level exec function.
String	getFinal()
String	getInitial()
String	getIntermed()
org.apache.pig.data.DataByteArray	getValue() Returns the result that has been built up by multiple calls to accumulate(org.apache.pig.data.Tuple).

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLoadCaster, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, needEndOfAllInputProcessing, outputSchema, progress, setEndOfAllInput, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataToSketch

public DataToSketch()

Default constructor. Assumes default k.

DataToSketch

public DataToSketch(String kStr)

String constructor.

Parameters:

kStr - string representation of k

Method Detail

exec

public org.apache.pig.data.DataByteArray exec(org.apache.pig.data.Tuple inputTuple)
                                       throws IOException

Top-level exec function. This method accepts an input Tuple containing a Bag of one or more inner Datum Tuples and returns a single updated Sketch as a DataByteArray.

Types are in the form: Java data type: Pig DataType

Input Tuple

• Tuple: TUPLE (Must contain only one field)
  • index 0: DataBag: BAG (May contain 0 or more Inner Tuples)
    • index 0: Tuple: TUPLE Datum Tuple
    • ...
    • index n-1: Tuple: TUPLE Datum Tuple

Datum Tuple

• Tuple: TUPLE (Must contain only one field)
  • index 0: Float: FLOAT

Sketch Tuple

• Tuple: TUPLE (Contains exactly 1 field)
  • index 0: DataByteArray: BYTEARRAY = a serialized KllFloatsSketch object.

Specified by:

exec in class org.apache.pig.EvalFunc<org.apache.pig.data.DataByteArray>

Parameters:

inputTuple - A tuple containing a single bag, containing Datum Tuples

Returns:

serialized sketch

Throws:

IOException - from Pig

See Also:

"org.apache.pig.EvalFunc.exec(org.apache.pig.data.Tuple)"

accumulate

public void accumulate(org.apache.pig.data.Tuple inputTuple)
                throws IOException

An Accumulator version of the standard exec() method. Like exec(), accumulator is called with a bag of Datum Tuples. Unlike exec(), it doesn't serialize the sketch at the end. Instead, it can be called multiple times, each time with another bag of Datum Tuples.

Specified by:

accumulate in interface org.apache.pig.Accumulator<org.apache.pig.data.DataByteArray>

Parameters:

inputTuple - A tuple containing a single bag, containing Datum Tuples.

Throws:

IOException - by Pig

See Also:

exec(org.apache.pig.data.Tuple), "org.apache.pig.Accumulator.accumulate(org.apache.pig.data.Tuple)"

getValue

public org.apache.pig.data.DataByteArray getValue()

Returns the result that has been built up by multiple calls to accumulate(org.apache.pig.data.Tuple).

Specified by:

getValue in interface org.apache.pig.Accumulator<org.apache.pig.data.DataByteArray>

Returns:

serialized sketch

See Also:

"org.apache.pig.Accumulator.getValue()"

cleanup

public void cleanup()

Cleans up the UDF state after being called using the Accumulator interface.

Specified by:

cleanup in interface org.apache.pig.Accumulator<org.apache.pig.data.DataByteArray>

See Also:

"org.apache.pig.Accumulator.cleanup()"

getInitial

public String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

getIntermed

public String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

getFinal

public String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic