Package org.apache.lucene.util.hnsw

Class HnswGraphBuilder

java.lang.Object

org.apache.lucene.util.hnsw.HnswGraphBuilder

All Implemented Interfaces:

HnswBuilder

Direct Known Subclasses:

InitializedHnswGraphBuilder, MergingHnswGraphBuilder

public class HnswGraphBuilder
extends Object
implements HnswBuilder

Builder for HNSW graph. See HnswGraph for a gloss on the algorithm and the meaning of the hyper-parameters.

Thread-safety: This class is NOT thread safe, it cannot be shared across threads, however, it IS safe for multiple HnswGraphBuilder to build the same graph, if the graph's size is known in the beginning (like when doing merge)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	HnswGraphBuilder.GraphBuilderKnnCollector	A restricted, specialized knnCollector that can be used when building a graph.

Field Summary

Fields

Modifier and Type	Field	Description
protected final HnswGraphBuilder.GraphBuilderKnnCollector	beamCandidates
static final int	DEFAULT_BEAM_WIDTH	Default number of the size of the queue maintained while searching during a graph construction.
static final int	DEFAULT_MAX_CONN	Default number of maximum connections per node
protected boolean	frozen
protected final HnswGraphSearcher	graphSearcher
protected final OnHeapHnswGraph	hnsw
static final String	HNSW_COMPONENT	A name for the HNSW component for the info-stream *
protected final org.apache.lucene.util.hnsw.HnswLock	hnswLock
protected InfoStream	infoStream
protected final int	M
static long	randSeed	Random seed for level generation; public to expose for testing *
protected final UpdateableRandomVectorScorer	scorer

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth, long seed, int graphSize)	Reads all the vectors from vector values, builds a graph connecting them by their dense ordinals, using the given hyperparameter settings, and returns the resulting graph.
protected	HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth, long seed, OnHeapHnswGraph hnsw)
protected	HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth, long seed, OnHeapHnswGraph hnsw, org.apache.lucene.util.hnsw.HnswLock hnswLock, HnswGraphSearcher graphSearcher)	Reads all the vectors from vector values, builds a graph connecting them by their dense ordinals, using the given hyperparameter settings, and returns the resulting graph.
protected	HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier, int beamWidth, long seed, OnHeapHnswGraph hnsw)

Method Summary

Modifier and Type	Method	Description
void	addGraphNode(int node)	Inserts a doc with a vector value to the graph
void	addGraphNode(int node, IntHashSet eps0)	Inserts a doc with a vector value to the graph, searching on level 0 with provided entry points
protected void	addVectors(int minOrd, int maxOrd)	add vectors in range [minOrd, maxOrd)
OnHeapHnswGraph	build(int maxOrd)	Adds all nodes to the graph up to the provided maxOrd.
static HnswGraphBuilder	create(RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth, long seed)
static HnswGraphBuilder	create(RandomVectorScorerSupplier scorerSupplier, int M, int beamWidth, long seed, int graphSize)
OnHeapHnswGraph	getCompletedGraph()	Once this method is called, no further updates to the graph are accepted (addGraphNode will throw IllegalStateException).
OnHeapHnswGraph	getGraph()
void	setInfoStream(InfoStream infoStream)	Set info-stream to output debugging information

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MAX_CONN

public static final int DEFAULT_MAX_CONN

Default number of maximum connections per node

See Also:

• Constant Field Values

DEFAULT_BEAM_WIDTH

public static final int DEFAULT_BEAM_WIDTH

Default number of the size of the queue maintained while searching during a graph construction.

See Also:

• Constant Field Values

HNSW_COMPONENT

public static final String HNSW_COMPONENT

A name for the HNSW component for the info-stream *

See Also:

• Constant Field Values

randSeed

public static long randSeed

Random seed for level generation; public to expose for testing *

M

protected final int M

scorer

protected final UpdateableRandomVectorScorer scorer

graphSearcher

protected final HnswGraphSearcher graphSearcher

beamCandidates

protected final HnswGraphBuilder.GraphBuilderKnnCollector beamCandidates

hnsw

protected final OnHeapHnswGraph hnsw

hnswLock

protected final org.apache.lucene.util.hnsw.HnswLock hnswLock

infoStream

protected InfoStream infoStream

frozen

protected boolean frozen

Constructor Details

HnswGraphBuilder

protected HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier,
 int M,
 int beamWidth,
 long seed,
 int graphSize)
                    throws IOException

Reads all the vectors from vector values, builds a graph connecting them by their dense ordinals, using the given hyperparameter settings, and returns the resulting graph.

Parameters:

scorerSupplier - a supplier to create vector scorer from ordinals.

M - – graph fanout parameter used to calculate the maximum number of connections a node can have – M on upper layers, and M * 2 on the lowest level.

beamWidth - the size of the beam search to use when finding nearest neighbors.

seed - the seed for a random number generator used during graph construction. Provide this to ensure repeatable construction.

graphSize - size of graph, if unknown, pass in -1

Throws:

IOException

HnswGraphBuilder

protected HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier,
 int M,
 int beamWidth,
 long seed,
 OnHeapHnswGraph hnsw)
                    throws IOException

Throws:

IOException

HnswGraphBuilder

protected HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier,
 int beamWidth,
 long seed,
 OnHeapHnswGraph hnsw)
                    throws IOException

Throws:

IOException

HnswGraphBuilder

protected HnswGraphBuilder(RandomVectorScorerSupplier scorerSupplier,
 int M,
 int beamWidth,
 long seed,
 OnHeapHnswGraph hnsw,
 org.apache.lucene.util.hnsw.HnswLock hnswLock,
 HnswGraphSearcher graphSearcher)
                    throws IOException

Reads all the vectors from vector values, builds a graph connecting them by their dense ordinals, using the given hyperparameter settings, and returns the resulting graph.

Parameters:

scorerSupplier - a supplier to create vector scorer from ordinals.

M - – graph fanout parameter used to calculate the maximum number of connections a node can have – M on upper layers, and M * 2 on the lowest level.

beamWidth - the size of the beam search to use when finding nearest neighbors.

seed - the seed for a random number generator used during graph construction. Provide this to ensure repeatable construction.

hnsw - the graph to build, can be previously initialized

Throws:

IOException

Method Details

create

public static HnswGraphBuilder create(RandomVectorScorerSupplier scorerSupplier,
 int M,
 int beamWidth,
 long seed)
                               throws IOException

Throws:

IOException

create

public static HnswGraphBuilder create(RandomVectorScorerSupplier scorerSupplier,
 int M,
 int beamWidth,
 long seed,
 int graphSize)
                               throws IOException

Throws:

IOException

build

public OnHeapHnswGraph build(int maxOrd)
                      throws IOException

Description copied from interface: HnswBuilder

Adds all nodes to the graph up to the provided maxOrd.

Specified by:

build in interface HnswBuilder

Parameters:

maxOrd - The maximum ordinal (excluded) of the nodes to be added.

Throws:

IOException

setInfoStream

public void setInfoStream(InfoStream infoStream)

Description copied from interface: HnswBuilder

Set info-stream to output debugging information

Specified by:

setInfoStream in interface HnswBuilder

getCompletedGraph

public OnHeapHnswGraph getCompletedGraph()
                                  throws IOException

Description copied from interface: HnswBuilder

Once this method is called, no further updates to the graph are accepted (addGraphNode will throw IllegalStateException). Final modifications to the graph (eg patching up disconnected components, re-ordering node ids for better delta compression) may be triggered, so callers should expect this call to take some time.

Specified by:

getCompletedGraph in interface HnswBuilder

Throws:

IOException

getGraph

public OnHeapHnswGraph getGraph()

Specified by:

getGraph in interface HnswBuilder

addVectors

protected void addVectors(int minOrd,
 int maxOrd)
                   throws IOException

add vectors in range [minOrd, maxOrd)

Throws:

IOException

addGraphNode

public void addGraphNode(int node)
                  throws IOException

Description copied from interface: HnswBuilder

Inserts a doc with a vector value to the graph

Specified by:

addGraphNode in interface HnswBuilder

Throws:

IOException

addGraphNode

public void addGraphNode(int node,
 IntHashSet eps0)
                  throws IOException

Description copied from interface: HnswBuilder

Inserts a doc with a vector value to the graph, searching on level 0 with provided entry points

Specified by:

addGraphNode in interface HnswBuilder

Throws:

IOException