org.eclipse.january.dataset

Class LazyDataset

java.lang.Object

org.eclipse.january.dataset.LazyDatasetBase

org.eclipse.january.dataset.LazyDataset

All Implemented Interfaces:

Serializable, Cloneable, ILazyDataset, IMetadataProvider, INameable

Direct Known Subclasses:

LazyDynamicDataset

public class LazyDataset
extends LazyDatasetBase
implements Serializable, Cloneable

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	begSlice
protected int[]	delSlice
protected int	isize
protected ILazyLoader	loader
protected int[]	map
protected Map<Class<? extends MetadataType>,List<MetadataType>>	oMetadata
protected int[]	oShape
protected int[]	padding
protected long	size
protected int[]	sShape

Fields inherited from class org.eclipse.january.dataset.LazyDatasetBase

catchExceptions, logger, metadata, name, shape

Constructor Summary

Constructors

Constructor and Description
LazyDataset(ILazyLoader loader, String name, Class<? extends Dataset> clazz, int... shape) Create a lazy dataset
LazyDataset(ILazyLoader loader, String name, int elements, Class<? extends Dataset> clazz, int... shape) Create a lazy dataset
LazyDataset(String name, int dtype, int[] shape, ILazyLoader loader) Deprecated. Use LazyDataset(ILazyLoader, String, int, Class, int[])
LazyDataset(String name, int dtype, int elements, int[] shape, ILazyLoader loader) Deprecated. Use LazyDataset(ILazyLoader, String, int, Class, int[])

Method Summary

Modifier and Type	Method and Description
protected SliceND	calcTrueSlice(SliceND slice) Calculate absolute slice
LazyDataset	clone() Clone dataset
static LazyDataset	createLazyDataset(Dataset dataset) Create a lazy dataset based on in-memory data (handy for testing)
protected SliceND	createSlice(int[] nstart, int[] nstop, int[] nstep)
boolean	equals(Object obj)
int	getDType() Can return -1 for unknown
int	getElementsPerItem() Can return -1 for unknown
Class<? extends Dataset>	getInterface()
static int	getMaxSliceLength(ILazyDataset lazySet, int dimension) Gets the maximum size of a slice of a dataset in a given dimension which should normally fit in memory.
int	getSize() The size of the dataset is the number of items in the array
Dataset	getSlice(IMonitor monitor, int[] start, int[] stop, int[] step) Get a slice of the dataset.
Dataset	getSlice(IMonitor monitor, Slice... slice) Get a slice of the dataset.
Dataset	getSlice(IMonitor monitor, SliceND slice) Get a slice of the dataset.
Dataset	getSlice(int[] start, int[] stop, int[] step) Get a slice of the dataset.
Dataset	getSlice(Slice... slice) Get a slice of the dataset.
Dataset	getSlice(SliceND slice) Get a slice of the dataset.
LazyDataset	getSliceView(int[] start, int[] stop, int[] step) Get a slice of the dataset.
LazyDataset	getSliceView(Slice... slice) Get a slice of the dataset.
LazyDataset	getSliceView(SliceND slice) Get a slice of the dataset.
LazyDataset	getTransposedView(int... axes) Permute copy of dataset's axes so that given order is old order:
int	hashCode()
void	setShape(int... shape) Set a compatible shape for dataset.
LazyDataset	squeezeEnds() Remove dimensions of 1 from ends of shape of the dataset
String	toString()

Methods inherited from class org.eclipse.january.dataset.LazyDatasetBase

addMetadata, checkPermutatedAxes, clearMetadata, copyMetadata, copyMetadata, createFromSerializable, dirtyMetadata, findMetadataTypeSubInterfaces, getElementClass, getErrorMetadata, getErrors, getFirstMetadata, getMetadata, getMetadata, getMetadataMap, getName, getRank, getShape, hasErrors, reshapeMetadata, restoreMetadata, setDirty, setErrors, setMetadata, setName, sliceMetadata, transposeMetadata

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

oMetadata

protected Map<Class<? extends MetadataType>,List<MetadataType>> oMetadata

oShape

protected int[] oShape

size

protected long size

isize

protected int isize

loader

protected ILazyLoader loader

begSlice

protected int[] begSlice

delSlice

protected int[] delSlice

sShape

protected int[] sShape

Since:

2.2

padding

protected int[] padding

Since:

2.2

map

protected int[] map

Constructor Detail

LazyDataset

public LazyDataset(ILazyLoader loader,
                   String name,
                   int elements,
                   Class<? extends Dataset> clazz,
                   int... shape)

Create a lazy dataset

Parameters:

loader -

name -

elements -

clazz - dataset interface

shape -

Since:

2.3

LazyDataset

public LazyDataset(ILazyLoader loader,
                   String name,
                   Class<? extends Dataset> clazz,
                   int... shape)

Create a lazy dataset

Parameters:

loader -

name -

clazz - dataset interface

shape -

Since:

2.3

LazyDataset

@Deprecated
public LazyDataset(String name,
                               int dtype,
                               int elements,
                               int[] shape,
                               ILazyLoader loader)

Deprecated. Use LazyDataset(ILazyLoader, String, int, Class, int[])

Create a lazy dataset

Parameters:

name -

dtype - dataset type

elements -

shape -

loader -

LazyDataset

@Deprecated
public LazyDataset(String name,
                               int dtype,
                               int[] shape,
                               ILazyLoader loader)

Deprecated. Use LazyDataset(ILazyLoader, String, int, Class, int[])

Create a lazy dataset

Parameters:

name -

dtype - dataset type

shape -

loader -

Method Detail

createLazyDataset

public static LazyDataset createLazyDataset(Dataset dataset)

Create a lazy dataset based on in-memory data (handy for testing)

Parameters:

dataset -

getDType

public int getDType()

Can return -1 for unknown

Specified by:

getDType in class LazyDatasetBase

Returns:

type of dataset item

getInterface

public Class<? extends Dataset> getInterface()

Returns:

dataset interface that supports element class and number of elements

Since:

2.3

getElementsPerItem

public int getElementsPerItem()

Can return -1 for unknown

Specified by:

getElementsPerItem in interface ILazyDataset

Returns:

Number of elements per item

getSize

public int getSize()

Description copied from interface: ILazyDataset

The size of the dataset is the number of items in the array

Specified by:

getSize in interface ILazyDataset

Returns:

number of data items

toString

public String toString()

Overrides:

toString in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class LazyDatasetBase

equals

public boolean equals(Object obj)

Overrides:

equals in class LazyDatasetBase

clone

public LazyDataset clone()

Description copied from interface: ILazyDataset

Clone dataset

Specified by:

clone in interface ILazyDataset

Overrides:

clone in class LazyDatasetBase

Returns:

a (shallow) copy of dataset

setShape

public void setShape(int... shape)

Description copied from interface: ILazyDataset

Set a compatible shape for dataset. A shape is compatible if it has the capacity to contain the same number of items

Specified by:

setShape in interface ILazyDataset

squeezeEnds

public LazyDataset squeezeEnds()

Description copied from interface: ILazyDataset

Remove dimensions of 1 from ends of shape of the dataset

Specified by:

squeezeEnds in interface ILazyDataset

getSlice

public Dataset getSlice(int[] start,
                        int[] stop,
                        int[] step)
                 throws DatasetException

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned dataset is a copied selection of items

Specified by:

getSlice in interface ILazyDataset

Parameters:

start - specifies the starting indexes (can be null for origin)

stop - specifies the stopping indexes (can be null for end)

step - specifies the steps in the slice (can be null for unit steps)

Returns:

The dataset of the sliced data

Throws:

DatasetException

getSlice

public Dataset getSlice(Slice... slice)
                 throws DatasetException

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned dataset is a copied selection of items

Specified by:

getSlice in interface ILazyDataset

Parameters:

slice - an array of slice objects (the array can be null or contain nulls)

Returns:

The dataset of the sliced data

Throws:

DatasetException

getSlice

public Dataset getSlice(SliceND slice)
                 throws DatasetException

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned dataset is a copied selection of items

Specified by:

getSlice in interface ILazyDataset

Parameters:

slice - an n-D slice

Returns:

The dataset of the sliced data

Throws:

DatasetException

getSlice

public Dataset getSlice(IMonitor monitor,
                        Slice... slice)
                 throws DatasetException

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned dataset is a copied selection of items

Specified by:

getSlice in interface ILazyDataset

slice - an array of slice objects (the array can be null or contain nulls)

Returns:

The dataset of the sliced data

Throws:

DatasetException

getSliceView

public LazyDataset getSliceView(Slice... slice)

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned lazy dataset is a view on a selection of items

Specified by:

getSliceView in interface ILazyDataset

Parameters:

slice - an array of slice objects (the array can be null or contain nulls)

Returns:

The sliced view of a lazy dataset

getSliceView

public LazyDataset getSliceView(int[] start,
                                int[] stop,
                                int[] step)

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned lazy dataset is a view on a selection of items

Specified by:

getSliceView in interface ILazyDataset

Parameters:

start - specifies the starting indexes (can be null for origin)

stop - specifies the stopping indexes (can be null for end)

step - specifies the steps in the slice (can be null for unit steps)

Returns:

The sliced view of a lazy dataset

getSliceView

public LazyDataset getSliceView(SliceND slice)

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned lazy dataset is a view on a selection of items

Specified by:

getSliceView in interface ILazyDataset

Parameters:

slice - an n-D slice

Returns:

The sliced view of a lazy dataset

getSlice

public Dataset getSlice(IMonitor monitor,
                        int[] start,
                        int[] stop,
                        int[] step)
                 throws DatasetException

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned dataset is a copied selection of items

Specified by:

getSlice in interface ILazyDataset

start - specifies the starting indexes (can be null for origin)

stop - specifies the stopping indexes (can be null for end)

step - specifies the steps in the slice (can be null for unit steps)

Returns:

The dataset of the sliced data

Throws:

DatasetException

getSlice

public Dataset getSlice(IMonitor monitor,
                        SliceND slice)
                 throws DatasetException

Description copied from interface: ILazyDataset

Get a slice of the dataset. The returned dataset is a copied selection of items

Specified by:

getSlice in interface ILazyDataset

slice - an n-D slice

Returns:

The dataset of the sliced data

Throws:

DatasetException

getTransposedView

public LazyDataset getTransposedView(int... axes)

Description copied from interface: ILazyDataset

Permute copy of dataset's axes so that given order is old order:

  axisPerm = (p(0), p(1),...) => newdata(n(0), n(1),...) = olddata(o(0), o(1), ...)
  such that n(i) = o(p(i)) for all i
 

I.e. for a 3D dataset (1,0,2) implies the new dataset has its 1st dimension running along the old dataset's 2nd dimension and the new 2nd is the old 1st. The 3rd dimension is left unchanged.

Specified by:

getTransposedView in interface ILazyDataset

Parameters:

axes - if zero length then axes order reversed

Returns:

remapped view of data

calcTrueSlice

protected final SliceND calcTrueSlice(SliceND slice)

Calculate absolute slice

Parameters:

slice -

Returns:

true slice or null if zero-sized

createSlice

protected SliceND createSlice(int[] nstart,
                              int[] nstop,
                              int[] nstep)

getMaxSliceLength

public static int getMaxSliceLength(ILazyDataset lazySet,
                                    int dimension)

Gets the maximum size of a slice of a dataset in a given dimension which should normally fit in memory. Note that it might be possible to get more in memory, this is a conservative estimate and seems to almost always work at the size returned; providing Xmx is less than the physical memory. To get more in memory increase -Xmx setting or use an expression which calls a rolling function (like rmean) instead of slicing directly to memory.

Parameters:

lazySet -

dimension -

Returns:

maximum size of dimension that can be sliced.