Microsoft.ML.OnnxRuntime
Return immutable collection of results
This is a legacy class that is kept for backward compatibility.
Use OrtValue based API.
This class serves as a container for model run output values including
tensors, sequences of tensors, sequences and maps.
The class must be disposed of.
It disposes of _ortValueHolder that owns the underlying Ort output value and
anything else that would need to be disposed by the instance of the class.
Use factory method CreateFromOrtValue to obtain an instance of the class.
Ctor
Name of the output value
Managed object created to represent output value, such as DenseTensor{T};
List or Dictionary
Tensor element type if value type is a Tensor
Object that holds native resources.
Typically, this is an output OrtValue that holds native memory where Tensor is mapped but may also be
other things that would need to be disposed by this instance depending on how IOrtValueOwner is implemented.
Ctor for non-tensor values
Construct an instance that would contain a map in a form of a Dictionary
Currently a limited number of primitive types are supported as map keys and values.
So this is not a full implementation of the map type.
Only valid if ValueType is Tensor
Overrides the base class method. With respect to memoryHolder, it has no operation
to do, as this class maintains a native buffer via _ortValueHolder and the memory will be
disposed by it. This is the case when we are dealing with an OrtValue that is backed by native memory
and not by pinned managed memory.
This class is generally used for outputs to be created on top of the output OrtValue,
but the interface (derived from NamedOnnxValue) allows it to be passed as output and one of the test
cases does it. Unless we deprecate and re-do the interface, we must support it.
always set to null
Native OrtValue handle
Generally, this class is created on top of the values that are returned by the model run.
However, there is a test case that uses this value for output
It will return the OrtValue that was previously created, since the caller must understand what they are doing.
This function takes ortValue and constructs an instance of DisposableNamedOnnxValue.
The new instance takes ownership of the OrtValue and will dispose of it when it is disposed of.
becomes null on success.
an instance of DisposableNamedOnnxValue
This function takes ortValue and constructs an instance of DisposableNamedOnnxValue.
The new instance takes ownership of the OrtValue and will dispose of it when it is disposed of.
becomes null on success.
an instance of DisposableNamedOnnxValue
Creates an instance of DisposableNamedOnnxValue and takes ownership of ortValue.
on success.
name of the value
Underlying OrtValue. This becomes null on successful return.
This method creates an instance of DisposableNamedOnnxValue that has possession of ortValueElement
native memory Tensor and returns it to the caller.
data type
name of the output
native tensor. Becomes null on successful return.
DisposableNamedOnnxValue instance
This method will create an instance of DisposableNamedOnnxValue that will own ortSequenceValue
an all disposable native objects that are elements of the sequence
ortValueElement that has native sequence
used allocator
DisposableNamedOnnxValue
Will extract keys and values from the map and create a DisposableNamedOnnxValue from it
name of the output
ortValue that represents a map. Becomes null on success
DisposableNamedOnnxValue
This method maps keys and values of the map and copies them into a managed Dictionary
and returns as an instance of DisposableNamedOnnxValue. The method takes possession of ortValueMap,
ortValueTensorKeys and ortValueTensorValues and disposes of them.
becomes null on success return
keys shape in ints
becomes null on success
values shape in ints
becomes null on success
IDisposable implementation
true if invoked by Dispose()
IDisposable implementation
Enum conresponding to native onnxruntime error codes. Must be in sync with the native API
The Exception that is thrown for errors related ton OnnxRuntime
This is a legacy class that is kept for backward compatibility.
Use OrtValue based API.
Represents an OrtValue with its underlying buffer pinned
Creates a object from the tensor and pins its underlying buffer.
a disposable instance of FixedBufferOnnxValue
This is a factory method that creates a disposable instance of FixedBufferOnnxValue
on top of a buffer. Internally, it will pin managed buffer and will create
an OrtValue containing a tensor that will not own the memory.
Such instance of FixedBufferOnnxValue can be used both as input and output in InferenceSession.Run()
overload. As compared to CreateFromTensor(), this allows you to pass in buffers with custom data types
that are blittable as defined in https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types
I.e. those that have the same binary representation as the original type. This includes all existing types
but may also allow using custom types for Float16 and BFloat16 providing they have the same layout and size.
The resulting instance must be disposed of to release pinned memory and deallocate native OrtValue
See example below.
Blittable data type, compatible with supported types
memoryInfo. For managed buffers simply use OrtMemoryInfo.DefaultInstance
TensorElementType
shape of the tensor to be created
size of the allocation in bytes
a disposable instance of FixedBufferOnnxValue
Here is an example of using a 3rd party library class for processing float16/bfloat16.
Currently, to pass tensor data and create a tensor one must copy data to Float16/BFloat16 structures
so DenseTensor can recognize it.
If you are using a library that has a class Half and it is blittable, that is its managed in memory representation
matches native one and its size is 16-bits, you can use the following conceptual example
to feed/fetch data for inference using Half array. This allows you to avoid copying data from your Half[] to Float16[]
\code{.cs}
unsafe { Debug.Assert(sizeof(ushort) == sizeof(Half)); }
Half[] input = new Half[] { 5646, 12345 };
var input_shape = new long[] {input.Length};
Half[] output = new Half[40]; // Whatever the expected len/shape is must match
var output_shape = new long[] {output.Length};
var memInfo = OrtMemoryInfo.DefaultInstance; // CPU
using(var fixedBufferInput = FixedBufferOnnxvalue.CreateFromMemory{Half}(memInfo,
input, TensorElementType.Float16, input_shape, input.Length * sizeof(ushort))
using(var fixedBufferOutput = FixedBufferOnnxvalue.CreateFromMemory{Half}(memInfo,
output, TensorElementType.Float16, output_shape, output.Length * sizeof(ushort))
{
FixedBufferOnnxvalue[] inputValues = new FixedBufferOnnxvalue[]{fixedBufferInput};
FixedBufferOnnxvalue[] outputValues = new FixedBufferOnnxvalue[]{fixedBufferOutput};
session.Run(inputNames, inputValues, outputNames, outputValues);
// Output is now in output[]
}
\endcode
IDisposable implementation
true if invoked from Dispose()
IDisposable implementation
Represents an Inference Session on an ONNX Model.
This is a IDisposable class and it must be disposed of
using either a explicit call to Dispose() method or
a pattern of using() block. If this is a member of another
class that class must also become IDisposable and it must
dispose of InferenceSession in its Dispose() method.
A pointer to a underlying native instance of OrtSession
Dictionary that represents input metadata
Ordered list of input names
Dictionary that represent output metadata
Ordered list of output names
Dictionary that represents overridableInitializers metadata
This list holds Utf-8 converted input/output names allocated from a native heap
and as such do not require pinning. It must be disposed of (freed).
Introduced to reduce the GC burden as the names are used in every Run() call.
Constructs an InferenceSession from a model file
Constructs an InferenceSession from a model file and it will use
the provided pre-packed weights container to store and share pre-packed buffers
of shared initializers across sessions if any.
Model path
Instance of PrepackedWeightsContainer.
Lifetime of 'prepackedWeightsContainer' must be
managed by the user and it must outlive any sessions reliant on it
Constructs an InferenceSession from a model file, with some additional session options
Constructs an InferenceSession from a model file, with some additional session options
and it will use the provided pre-packed weights container to store and share pre-packed buffers
of shared initializers across sessions if any.
Model path
Session options
Instance of PrepackedWeightsContainer.
Lifetime of 'prepackedWeightsContainer' must be
managed by the user and it must outlive any sessions reliant on it
Constructs an InferenceSession from a model data in byte array
Constructs an InferenceSession from a model data (in byte array) and it will use
the provided pre-packed weights container to store and share pre-packed buffers
of shared initializers across sessions if any.
Model as byte array
Instance of PrepackedWeightsContainer.
Lifetime of 'prepackedWeightsContainer' must be
managed by the user and it must outlive any sessions reliant on it
Constructs an InferenceSession from a model data in byte array, with some additional session options
Constructs an InferenceSession from a model data (in byte array) with some additional
session options and it will use the provided pre-packed weights container to store
and share pre-packed buffers of shared initializers across sessions if any.
Model as byte array
Session Options
Instance of PrepackedWeightsContainer.
Lifetime of 'prepackedWeightsContainer' must be
managed by the user and it must outlive any sessions reliant on it
Meta data regarding the input nodes, keyed by input names
Ordered list of input names that can be accessed by index;
Metadata regarding the output nodes, keyed by output names
Ordered list of output names that can be accessed by index.
Metadata regarding the overridable initializers, keyed by node names
Runs the loaded model for the given inputs, and fetches all the outputs.
specify a collection of that indicates the input values.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Runs the loaded model for the given inputs, and fetches the outputs specified in .
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names to fetch.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Runs the loaded model for the given inputs, and fetches the specified outputs in . Uses the given RunOptions for this run.
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names to fetch.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Runs the loaded model for the given inputs, and fetches all the outputs.
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Runs the loaded model for the given inputs, and fetches the outputs specified in .
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names to fetch.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Runs the loaded model for the given inputs, and fetches the specified outputs in . Uses the given RunOptions for this run.
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names to fetch.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to accept the fetched data.
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names. Should match .
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to accept the fetched data.
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names. Should match .
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to receive the fetched data.
Specify a collection of that indicates the input values.
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to receive the fetched data.
Specify a collection of that indicates the input values.
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to receive the fetched data.
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names. Should match .
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to receive the fetched data.
Specify a collection of that indicates the input values.
Specify a collection of string that indicates the output names. Should match .
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to receive the fetched data.
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Specify a collection of that indicates the output values.
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to receive the fetched data.
Specify a collection of string that indicates the input names. Should match .
Specify a collection of that indicates the input values.
Specify a collection of that indicates the output values.
The API runs the inference taking a collection of OrtValues as input and
returning a collection of output OrtValues.
runOptions
A collection of input names.
To supply all names, use InputNames property
Input OrtValues. The size of the collection must match the size and the order of the inputNames
Output names requested. To supply all names, use OutputNames property.
A disposable collection of disposable OrtValues
This API takes inputs as a dictionary of input names paired with input OrtValues
It returns a disposable collection of OrtValues for outputs that were designated by outputNames
Dictionary of name/value pairs
requested outputs. To request all outputs, use OutputNames property of this sessions
A disposable collection of outputs
The API takes collections of inputNames/inputValues and collections of outputNames/outputValues.
The sizes of the corresponding collections must match.
The output OrtValues are pre-allocated and the API will fill the data into the OrtValues.
These MUST be tensors. The API does not support non-tensor types for output values.
The API is useful when the output values are tensors and their shapes are known, and you
prefer the output to go to the pre-allocated memory. In such a case, you create
output OrtValues over those pre-allocated buffers and pass them to the API.
runOptions, if null the defaults are used
collection of input names.
collection of input OrtValues. Must match the order and the number of input names.
Requested output names.
Pre-allocated output values.
The order and the number must match the specified output names. Shapes must match actual output values.
Create OrtIoBinding instance to bind pre-allocated buffers
to input/output
A new instance of OrtIoBinding
This method runs inference on the OrtIoBinding instance
The method does not return anything. This is a lightweight version of
RunWithBindingAndNames(). When you bind pre-allocated buffers to the output values
you may not want to fetch the outputs since you already have access to them so you can spare
the expense of fetching them and pairing with names.
You can still fetch the outputs by calling OrtIOBinding.GetOutputValues()
runOptions
ioBinding instance to use
This method runs inference on the OrtIoBinding instance. It returns a collection of OrtValues.
This method is useful when it is impossible to bind outputs to pre-allocated buffers, because
the output shape is not known in advance. In this case, the OrtValues returned by this method
are allocated and owned by ORT. The caller is responsible for disposing the collection.
RunOptions
Binding instance
A disposable collection of OrtValues
This method return a collection of DisposableNamedOnnxValue as in other interfaces
Query names from OrtIoBinding object and pair then with the array of OrtValues returned
from OrtIoBinding.GetOutputValues().
This API will be deprecated in favor of the API that returns a collection of OrtValues.
RunOptions
OrtIoBinding instance with bindings
optional parameter. If you already know the names of the outputs you can save a native
call to retrieve output names. They will be paired with the returned OrtValues and combined into DisposbleNamedOnnxValues.
Otherwise, the method will retrieve output names from the OrtIoBinding instance.
It is an error if you supply a different number of names than the returned outputs
A disposable collection of DisposableNamedOnnxValue that encapsulate output OrtValues
Ends profiling for the session.
Returns the profile file name.
Checks if the name is a known input or overridable initializer name
and if so, returns metadata for it.
metadata
NodeMetadata for the nodeName
Checks if the nodeName is a known output name and if so returns metadata for it.
Fetches/creates OrtValue for the content of the input
Fetches/Creates OrtValue for output
May return null if the onnx value type does not support pre-creation of output OrtValues
Run helper
names to convert to zero terminated utf8 and pin
extractor functor that helps extracting names from inputs
inputs/outputs metadata
This function obtains ortValues for NamedOnnxValue.
The problem with NamedOnnxValue is that it is not disposable and can not contain any disposable items.
so calling InputToOrtValue creates a new instance of OrtValue that needs to be disposed.
The deriving object DisposableNamedValue actually contains and owns OrtValue and it returns
it.
a collection of NamedOnnxValues
Metadata lookup function (input/initializers/output)
This property queries model metadata, constructs
an instance of ModelMetadata and caches it
Instance of ModelMetdata
Return the nanoseconds of profiling's start time
On some platforms, this timer may not be as precise as nanoseconds
For instance, on Windows and MacOS, the precision will be ~100ns
Run inference asynchronous in a thread of intra-op thread pool
run option, can be null
name of inputs
input ort values
name of outputs
output of ort values
task to be awaited
Initializes the session object with a native session handle
Value of a native session object
Other classes access
Finalizer. to cleanup session in case it runs
and the user forgets to Dispose() of the session
IDisposable implementation
IDisposable implementation
true if invoked from Dispose() method
This function is also used on failure in the constructor
Represents tensor element type and its shapes
Tensor Element type
TensorElementType enum
Shape
Array of dimensions
Symbolic dimensions
Array of symbolic dimensions if present.
Tensor element metadata
Represents sequnce metdata
__ctor
Element Metatada, recursive definition with a Tensor being a base case
may contain maps, tensors and other sequences
The class contains metadata for an optional input/output
__ctor
Element Metatada, recursive definition with a Tensor being a base case
may contain maps, tensors and sequences
Represents Map MetaData.
Key is always a tensor denoted by an element type
with value type being a recursive structure that may
contain other maps, sequences or tensors.
Key tensor data type
A value of TensorElementType enum
Value metadata
/// Instance of Nodemetadata for the value of the map
Resembles type and shape information of session-graph nodes, used for communicating the shape/type of input/output nodes
Constructs NodeMetadata for tensor
either ONNX_TYPE_TENSOR or ONNX_TYPE_SPARSETENSOR
Tensor type and shape information
__ctor for map metadata
__ctor for sequence metadata
__ctor
Retrieves MapMetadata, valid only if this node represents a Map.
when the instance does not contain map metadata
Retrieves SequenceMetadata, valid only if this node represents a Sequence
when the instance does not contain sequence metadata
Retrieves Optional type metadata, valid if this node is optional
Optional metadata is nothing more than just a container for all the usual
element types.
Type value of the node
A value of OnnxValueType enum
Node name in the natively allocated memory.
Present only on the top-level instance
metadata dictionary entries.
Avoids repeated conversion and pinning
This memory chunk is owned and freed by the InferenceSession
object.
Tensor shape valid only if this is a Tensor.
Preserved for API compatibility
Array of dimensions
Symbolic dimensions valid only if this is a Tensor.
Preserved for API compatibility
Array of symbolic dimensions if present.
.NET type that corresponds to the primitive Tensor data type.
Valid only if this is a Tensor.
System.Type
Tensor Element Type. Valid if tensor
Convinience method to check for string
Whether it is a Tensor
currently always returns true
A class that queries and caches model metadata and exposes
it as properties
Producer name string
producer name string
Graph name for this model
graph name string
Domain for this model
domain name string
Unstructured model description
description string
Unstructured graph description
description string
Version number
long version integer
Custom metadata key/value pairs
An instance of a Dictionary{string,string}
The class helps to feed the NamedOnnxValue as inference input.
It projects managed classes to OrtValues so they can be consumed
by the native onnxruntime library. if possible, it will avoid copying data.
The NamedOnnxValue can be a tensor, sequence or map.
For recursive structures, create nested NamedOnnxValue instances.
For example, a sequence instance would contain a list of NamedOnnxValue instances
that in turn may represent tensors or other ONNX values.
Dispatches the creation of the projection
OrtValue created according to the metadata
The function creates OrtValue objects for each element of the sequence
and then creates an OrtValue for the whole sequence.
NamedOnnxValue containing a IEnumerable{NamedOnnxValue}
sequence metadata
OrtValue that represents a sequence
Creates map projection. Since we support only primitive types in maps
we map two tensors (keys and values)
OrtValue
This pins memory that is contained within DenseTensor.
NodeOnnxValue containing DenseTensor
The class holds keys and values for the dictionary
in a for of two DenseTensors. The class is used to avoid
data copy and make these available to the native code.
Strings require special handling.
This is a legacy class that is kept for backward compatibility.
Use OrtValue based API.
The class associates a name with an Object.
The name of the class is a misnomer, it does not hold any Onnx values,
just managed representation of them.
The class is currently used as both inputs and outputs. Because it is non-
disposable, it can not hold on to any native objects.
When used as input, we temporarily create OrtValues that map managed inputs
directly. Thus we are able to avoid copying of contiguous data.
For outputs, tensor buffers works the same as input, providing it matches
the expected output shape. For other types (maps and sequences) we create a copy of the data.
This is because, the class is not Disposable and it is a public interface, thus it can not own
the underlying OrtValues that must be destroyed before Run() returns.
To avoid data copying on output, use DisposableNamedOnnxValue class that is returned from Run() methods.
This provides access to the native memory tensors and avoids copying.
It is a recursive structure that may contain Tensors (base case)
Other sequences and maps. Although the OnnxValueType is exposed,
the caller is supposed to know the actual data type contained.
The convention is that for tensors, it would contain a DenseTensor{T} instance or
anything derived from Tensor{T}.
For sequences, it would contain a IList{T} where T is an instance of NamedOnnxValue that
would contain a tensor or another type.
For Maps, it would contain a IDictionary{K, V} where K,V are primitive types or strings.
Managed Tensor, Dictionary or IList
Name of the instance, model input/output
Constructs an instance of NamedOnnxValue and represents
a model input to an inference session.
input/output name
Object that may be a tensor, Dictionary, IList
Constructs an instance that contains a tensor, sequence or optional type.
Use this to construct maps
Onnx Value Type if known. In general, NamedOnnxValue is able to contain
arbitrary objects. Please, follow the convention described in the class doc.
This is a factory method that instantiates NamedOnnxValue
and associated name with an instance of a Tensor
name
Tensor
This is a factory method that instantiates NamedOnnxValue.
It would contain a sequence of elements
Instantiates NamedOnnxValue that contains IDictionary{K, V}
Keys type
Values type
new instance of NamedOnnxValue
Exposes the name of the of the model input/output
name string
Exposes the underlying managed object
object
Try-get value as a Tensor<T>.
Type
Tensor object if contained value is a Tensor. Null otherwise
Try-get value as an Enumerable<T>.
T is usually a NamedOnnxValue instance that may contain
Tensors, Sequences, Maps or optional types
Type
Enumerable object if contained value is a Enumerable. Null otherwise
Try-get value as an Dictionary<K,V>.
Key type currently primitive type only
Value type, currently primitive type only
Dictionary object if contained value is a Dictionary. Null otherwise
Pin the underlying memory and create an instance of OrtValue containing a tensor
based on the pinned managed memory. The caller is responsible for Disposing
both OrtValue and pinnedMemoryHandle
dispose after returned OrtValue is disposed
The native OrtValue handle
Produces an output value for outputs. This produces an output value
only for tensors or optional types that can contain a tensor.
For all other Onnx value types, this method throws. Use Run() overloads
that return DisposableNamedOnnxValue to get access to all Onnx value types
that may be returned as output.
Checks the native Status if the errocode is OK/Success. Otherwise constructs an appropriate exception and throws.
Releases the native status object, as needed.
Creates native OrtTensorRTProviderOptions instance
(output) native instance of OrtTensorRTProviderOptions
Updates native OrtTensorRTProviderOptions instance using given key/value pairs
native instance of OrtTensorRTProviderOptions
configuration keys of OrtTensorRTProviderOptions
configuration values of OrtTensorRTProviderOptions
number of configuration keys
Get native OrtTensorRTProviderOptionsV2 in serialized string
instance of OrtAllocator
is a UTF-8 null terminated string allocated using 'allocator'
Releases native OrtTensorRTProviderOptions instance
native instance of OrtTensorRTProviderOptions to be released
Creates native OrtCUDAProviderOptions instance
(output) native instance of OrtCUDAProviderOptions
Updates native OrtCUDAProviderOptions instance using given key/value pairs
native instance of OrtCUDAProviderOptions
configuration keys of OrtCUDAProviderOptions
configuration values of OrtCUDAProviderOptions
number of configuration keys
Get native OrtCUDAProviderOptionsV2 in serialized string
instance of OrtAllocator
is a UTF-8 null terminated string allocated using 'allocator'
Releases native OrtCUDAProviderOptions instance
native instance of OrtCUDAProviderOptions to be released
Creates native OrtROCMProviderOptions instance
(output) native instance of OrtROCMProviderOptions
Updates native OrtROCMProviderOptions instance using given key/value pairs
native instance of OrtROCMProviderOptions
configuration keys of OrtROCMProviderOptions
configuration values of OrtROCMProviderOptions
number of configuration keys
Get native OrtROCMProviderOptions in serialized string
instance of OrtAllocator
is a UTF-8 null terminated string allocated using 'allocator'
Releases native OrtROCMProviderOptions instance
native instance of OrtROCMProviderOptions to be released
Creates an instance of OrtSession with provided parameters
Native OrtEnv instance
UTF-8 bytes corresponding to model string path
Native SessionOptions instance
Native OrtPrepackedWeightsContainer instance
(Output) Created native OrtSession instance
Creates an instance of OrtSession with provided parameters
Native OrtEnv instance
Byte array corresponding to the model
Size of the model in bytes
Native SessionOptions instance
Native OrtPrepackedWeightsContainer instance
(Output) Created native OrtSession instance
Add session config entry
Native SessionOptions instance
Config key
Config value
**
Append a TensorRT EP instance (configured based on given provider options) to the native OrtSessionOptions instance
Native OrtSessionOptions instance
Native OrtTensorRTProviderOptions instance
Append a TensorRT EP instance (configured based on given provider options) to the native OrtSessionOptions instance
Native OrtSessionOptions instance
Native OrtTensorRTProviderOptionsV2 instance
Append a CUDA EP instance (configured based on given provider options) to the native OrtSessionOptions instance
Native OrtSessionOptions instance
Native OrtCUDAProviderOptions instance
Append a CUDA EP instance (configured based on given provider options) to the native OrtSessionOptions instance
Native OrtSessionOptions instance
Native OrtCUDAProviderOptionsV2 instance
Append a ROCm EP instance (configured based on given provider options) to the native OrtSessionOptions instance
Native OrtSessionOptions instance
Native OrtROCMProviderOptions instance
Free Dimension override (by denotation)
Native SessionOptions instance
Dimension denotation
Dimension value
Free Dimension override (by name)
Native SessionOptions instance
Dimension name
Dimension value
Register custom op library
Native SessionOptions instance
Library path
(out) Native library handle
Register custom op library. ORT will manage freeing the library.
Native SessionOptions instance
Library path
Add initializer that is shared across Sessions using this SessionOptions (by denotation)
Native SessionOptions instance
Name of the initializer
Native OrtValue instance
Append an execution provider instance to the native OrtSessionOptions instance.
'SNPE', 'XNNPACK' and 'CoreML' are currently supported as providerName values.
The number of providerOptionsKeys must match the number of providerOptionsValues and equal numKeys.
Native OrtSessionOptions instance
Execution provider to add.
Configuration keys to add
Configuration values to add
Number of configuration keys
Memory maps the adapter file, wraps it into the adapter object
and returns it.
absolute path to the adapter file
optional device allocator or null
New LoraAdapter object
Creates LoraAdapter instance from a byte array that must
represents a valid LoraAdapter formst.
bytes
size in bytes
optional device allocator
resuling LoraAdapter instance
Add run config entry
Native RunOptions instance
Config key
Config value
Do not free the returned value
Create an instance of arena configuration which will be used to create an arena based allocator
See docs/C_API.md for details on what the following parameters mean and how to choose these values
Maximum amount of memory the arena allocates
Strategy for arena expansion
Size of the region that the arena allocates first
Maximum amount of fragmentation allowed per chunk
Pointer to a native OrtStatus instance indicating success/failure of config creation
Destroy an instance of an arena configuration instance
arena configuration instance to be destroyed
Create an instance of allocator according to mem_info
Session that this allocator should be used with
memory allocator specs
out pointer to a new allocator instance
Destroy an instance of an allocator created by OrtCreateAllocator
instance to be destroyed
Allocate a chunk of native memory
allocator instance
bytes to allocate
out pointer to the allocated memory. Must be freed by OrtAllocatorFree
Release native memory allocated by an allocator
allocator instance
pointer to native memory allocated by the allocator instance
Create OrtIoBinding instance that is used to bind memory that is allocated
either by a 3rd party allocator or an ORT device allocator. Such memory should be wrapped by
a native OrtValue of Tensor type. By binding such named values you will direct ORT to read model inputs
and write model outputs to the supplied memory.
session to create OrtIoBinding instance
out a new instance of OrtIoBinding
Destroy OrtIoBinding instance created by OrtCreateIoBinding
instance of OrtIoBinding
Bind OrtValue to the model input with the specified name
If binding with the specified name already exists, it will be replaced
instance of OrtIoBinding
model input name (utf-8)
OrtValue that is used for input (may wrap arbitrary memory).
The param instance is copied internally so this argument may be released.
The API calls Sync() on all EP providers present. This blocks until the device has completed
all preceding requested tasks. This is necessary when memory synchronization is required.
For example, the memory bound to an input is likely to be on a different CUDA stream.
For some scenarios and devices this may be a no-op, use
your best judgment.
instance of OrtIoBinding
An instance of OrtStatus or null
Bind OrtValue to the model output with the specified name
If binding with the specified name already exists, it will be replaced
instance of OrtIoBinding
model output name (utf-8)
OrtValue that is used for output (may wrap arbitrary memory).
The param instance is copied internally so this argument may be released.
Bind a device to the model output with the specified name
This is useful when the OrtValue can not be allocated ahead of time
due to unknown dimensions.
Instance of OrtIoBinding
UTF-8 zero terminated name
OrtMemoryInfo instance that contains device id. May be obtained from the device specific allocator instance
The API calls Sync() on all EP providers present. This blocks until the device has completed
all preceding requested tasks. This is necessary when memory synchronization is required.
For some scenarios and devices this may be a no-op, use your best judgment.
instance of OrtIoBinding
An instance of OrtStatus or null
The function will return all bound output names in the order they were bound.
It is the same order that the output values will be returned after RunWithBinding() is used.
The function will allocate two native allocations using the allocator supplied.
The caller is responsible for deallocating both of the buffers using the same allocator.
You may use OrtMemoryAllocation disposable class to wrap those allocations.
instance of OrtIoBinding
allocator to use for memory allocation
a continuous buffer that contains all output names.
Names are not zero terminated use lengths to extract strings. This needs to be deallocated.
A buffer that contains lengths (size_t) for each of the returned strings in order.
The buffer must be deallocated.
this contains the count of names returned which is the number of elements in lengths.
The function returns output values after the model has been run with RunWithBinding()
It returns a natively allocated buffer of OrtValue pointers. All of the OrtValues must be individually
released after no longer needed. You may use OrtValue disposable class to wrap the native handle and properly dispose it
in connection with DisposableList{T}. All values are returned in the same order as they were bound.
The buffer that contains OrtValues must deallocated using the same allocator that was specified as an argument.
You may use an instance OrtMemoryAllocation to properly dispose of the native memory.
instance of OrtIOBinding
allocator to use to allocate output buffer
allocated buffer that contains pointers (IntPtr) to individual OrtValue instances
count of OrtValues returned
Clears Input bindings. This is a convenience method.
Releasing OrtIoBinding instance would clear all bound inputs.
instance of OrtIoBinding
Clears Output bindings. This is a convenience method.
Releasing OrtIoBinding instance would clear all bound outputs.
instance of OrtIoBinding
Provides element-level access into a tensor.
instance of OrtIoBinding
Creates an allocator instance and registers it with the env to enable
sharing between multiple sessions that use the same env instance.
Lifetime of the created allocator will be valid for the duration of the environment.
Returns an error if an allocator with the same OrtMemoryInfo is already registered.
Native OrtEnv instance
Native OrtMemoryInfo instance
Native OrtArenaCfg instance
A pointer to native ortStatus indicating success/failure
Set the language projection for collecting telemetry data when Env is created
the source projected language
Gets the ModelMetadata associated with an InferenceSession
instance of OrtSession
(output) instance of OrtModelMetadata
Gets the producer name associated with a ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
(output) producer name from the ModelMetadata instance
Gets the graph name associated with a ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
(output) graph name from the ModelMetadata instance
Gets the domain associated with a ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
(output) domain from the ModelMetadata instance
Gets the description associated with a ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
(output) description from the ModelMetadata instance
Gets the description associated with a ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
(output) graph description from the ModelMetadata instance
Gets the version associated with a ModelMetadata instance
instance of OrtModelMetadata
(output) version from the ModelMetadata instance
Gets all the keys in the custom metadata map in the ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
(output) all keys in the custom metadata map
(output) number of keys in the custom metadata map
Gets the value associated with the given key in custom metadata map in the ModelMetadata instance
instance of OrtModelMetadata
instance of OrtAllocator
key in the custom metadata map
(output) value for the key in the custom metadata map
Frees ModelMetadata instance
instance of OrtModelMetadata
This function doesn't work with string tensor
this is a no-copy method whose pointer is only valid until the backing OrtValue* is free'd.
\param value A tensor created from OrtCreateTensor... function.
\param len total data length, not including the trailing '\0' chars.
\param value A tensor created from OrtCreateTensor... function.
\param index The index of the entry in the tensor to resize.
\param length_in_bytes Length to resize the string to.
\param buffer The resized buffer.
Get the symbolic dimension names for dimensions with a value of -1.
Order and number of entries is the same as values returned by GetDimensions.
The name may be empty for an unnamed symbolic dimension.
e.g.
If OrtGetDimensions returns [-1, -1, 2], OrtGetSymbolicDimensions would return an array with 3 entries.
If the values returned were ['batch', '', ''] it would indicate that
- the first dimension was a named symbolic dimension (-1 dim value and name in symbolic dimensions),
- the second dimension was an unnamed symbolic dimension (-1 dim value and empty string),
- the entry for the third dimension should be ignored as it is not a symbolic dimension (dim value >= 0).
How many elements does this tensor have.
May return a negative value
e.g.
[] -> 1
[1,3,4] -> 12
[2,0,4] -> 0
[-1,3,4] -> -1
Map Type API
Queries all the execution providers supported in the native onnxruntime shared library
(output) all execution providers (strings) supported in the native onnxruntime shared library
(output) number of execution providers (strings)
Releases all execution provider strings allocated and returned by OrtGetAvailableProviders
all execution providers (strings) returned by OrtGetAvailableProviders
number of execution providers (strings)
Create an instance of PrepackedWeightsContainer
(output) Created native OrtPrepackedWeightsContainer instance
Destroy an instance of PrepackedWeightsContainer
Native OrtPrepackedWeightsContainer instance to be destroyed
This helper class contains methods to create native OrtValue from a managed value object
Converts C# UTF-16 string to UTF-8 zero terminated
byte[] instance
string to be converted
UTF-8 encoded equivalent
This function converts the input string into UTF-8 encoding string (no zero termination)
straight into the pre-allocated native buffer. The buffer size
must match the required size and can be obtained in advance with
System.Text.Encoding.UTF8.GetByteCount(s).
fixed char* ptr
string length
Native buffer to write
Reads UTF-8 encode string from a C zero terminated string
and converts it into a C# UTF-16 encoded string
pointer to native or pinned memory where Utf-8 resides
optional allocator to free nativeUtf8 if it was allocated by OrtAllocator
Reads UTF-8 string from native C zero terminated string,
makes a copy of it on unmanaged heap and converts it to C# UTF-16 string,
then returns both C# string and the unmanaged copy of the UTF-8 string.
On return it deallocates the nativeUtf8 string using the specified allocator
allocator to use to free nativeUtf8
input
C# UTF-16 string
UTF-8 bytes in a unmanaged allocation, zero terminated
Converts C# UTF-16 string to UTF-8 zero terminated
byte[] instance
string to be converted
UTF-8 encoded equivalent
This class converts a string to a UTF8 encoded byte array and then copies it to an unmanaged buffer.
This is done, so we can pass it to the native code and avoid pinning.
Native allocation (UTF8-8 string length with terminating zero)
Actual native buffer
IDisposable implementation
Keeps a list of MarshaledString instances and provides a way to dispose them all at once.
It is a ref struct, so it can not be IDisposable.
Utility class used in SessioniOptions and ProviderOptions
A utility method to update the provider options, provides common functionality.
The actual key/value option pairs
to the object
encapsulates a native method that returns
Arg1=handle, Arg2=array of keys, Arg3=array of values, Arg4 - count, Arg5 - return ORT status
See documentation for OrtAllocatorType in C API
See documentation for OrtMemType in C API
This class encapsulates arena configuration information that will be used to define the behavior
of an arena based allocator
See docs/C_API.md for more details
Create an instance of arena configuration which will be used to create an arena based allocator
See docs/C_API.md for details on what the following parameters mean and how to choose these values
Maximum amount of memory the arena allocates
Strategy for arena expansion
Size of the region that the arena allocates first
Maximum amount of fragmentation allowed per chunk
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtEnv
always returns true
This class encapsulates and most of the time owns the underlying native OrtMemoryInfo instance.
Instance returned from OrtAllocator will not own OrtMemoryInfo, the class must be disposed
regardless.
Use this class to query and create OrtAllocator instances so you can pre-allocate memory for model
inputs/outputs and use it for binding. Instances of the class can also used to created OrtValues bound
to pre-allocated memory. In that case, the instance of OrtMemoryInfo contains the information about the allocator
used to allocate the underlying memory.
Default CPU based instance
Singleton instance of a CpuMemoryInfo
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
This allocator takes an native pointer to already existing
instance of OrtMemoryInfo. That instance may either be owned or not
owned. In the latter case, this class serves to expose native properties
of the instance.
Predefined utf8 encoded allocator names. Use them to construct an instance of
OrtMemoryInfo to avoid UTF-16 to UTF-8 conversion costs.
Predefined utf8 encoded allocator names. Use them to construct an instance of
OrtMemoryInfo to avoid UTF-16 to UTF-8 conversion costs.
Predefined utf8 encoded allocator names. Use them to construct an instance of
OrtMemoryInfo to avoid UTF-16 to UTF-8 conversion costs.
Predefined utf8 encoded allocator names. Use them to construct an instance of
OrtMemoryInfo to avoid UTF-16 to UTF-8 conversion costs.
Predefined utf8 encoded allocator names. Use them to construct an instance of
OrtMemoryInfo to avoid UTF-16 to UTF-8 conversion costs.
Create an instance of OrtMemoryInfo according to the specification
Memory info instances are usually used to get a handle of a native allocator
that is present within the current inference session object. That, in turn, depends
of what execution providers are available within the binary that you are using and are
registered with Add methods.
Allocator name. Use of the predefined above.
Allocator type
Device id
Memory type
Create an instance of OrtMemoryInfo according to the specification.
Allocator name
Allocator type
Device id
Memory type
Name of the allocator associated with the OrtMemoryInfo instance
Returns device ID
returns integer Id value
The below 2 are really properties but naming them is a challenge
as names would conflict with the returned type. Also, there are native
calls behind them so exposing them as Get() would be appropriate.
OrtMemoryType for the instance
Fetches allocator type from the underlying OrtAllocator
Returns allocator type
Overrides System.Object.Equals(object)
object to compare to
true if obj is an instance of OrtMemoryInfo and is equal to this
Compares this instance with another
OrtMemoryInfo to compare to
true if instances are equal according to OrtCompareMemoryInfo.
Overrides System.Object.GetHashCode()
integer hash value
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtMmeoryInfo
always returns true
This class represents an arbitrary buffer of memory
allocated and owned by the user. It can be either a CPU, GPU or other device memory
that can be suitably represented by IntPtr.
This is just a composite of the buffer related information.
The memory is assumed to be pinned if necessary and usable immediately
in the native code.
Constructor
use to accurately describe a piece of memory that this is wrapping
shape of this buffer
element type
the actual pointer to memory
size of the allocation in bytes
OrtMemoryInfo
Shape
Data type
Actual memory ptr
Size of the allocation in bytes
This class represents memory allocation made by a specific onnxruntime
allocator. Use OrtAllocator.Allocate() to obtain an instance of this class.
It implements IDisposable and makes use of the original allocator
used to allocate the memory. The lifespan of the allocator instance must eclipse the
lifespan of the allocation. Or, if you prefer, all OrtMemoryAllocation instances must be
disposed of before the corresponding allocator instances are disposed of.
This constructs an instance representing an native memory allocation.
Typically returned by OrtAllocator.Allocate(). However, some APIs return
natively allocated IntPtr using a specific allocator. It is a good practice
to wrap such a memory into OrtAllocation for proper disposal. You can set
size to zero if not known, which is not important for disposing.
Internal accessor to call native methods
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Size of the allocation
uint size of the allocation in bytes
Memory Information about this allocation
Returns OrtMemoryInfo from the allocator
Overrides SafeHandle.ReleaseHandle() to deallocate
a chunk of memory using the specified allocator.
always returns true
The class exposes native internal allocator for Onnxruntime.
This allocator enables you to allocate memory from the internal
memory pools including device allocations. Useful for binding.
Default CPU allocator instance
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Internal constructor wraps existing native allocators
Creates an instance of OrtAllocator according to the specifications in OrtMemorInfo.
The requested allocator should be available within the given session instance. This means
both, the native library was build with specific allocators (for instance CUDA) and the corresponding
provider was added to SessionsOptions before instantiating the session object.
OrtMemoryInfo instance owned by the allocator
Instance of OrtMemoryInfo describing this allocator
Allocate native memory. Returns a disposable instance of OrtMemoryAllocation
number of bytes to allocate
Instance of OrtMemoryAllocation
This internal interface is used for freeing memory.
pointer to a native memory chunk allocated by this allocator instance
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtAllocator
always returns true
Delegate for logging function callback.
Supply your function and register it with the environment to receive logging callbacks via
EnvironmentCreationOptions
Pointer to data passed into Constructor `log_param` parameter.
Log severity level.
Log category
Log Id.
Code location detail.
Log message.
Options you might want to supply when creating the environment.
Everything is optional.
Supply a log id to identify the application using ORT, otherwise, a default one will be used
Initial logging level so that you can see what is going on during environment creation
Default is LogLevel.Warning
Supply OrtThreadingOptions instance, otherwise null
Supply IntPtr logging param when registering logging function, otherwise IntPtr.Zero
This param will be passed to the logging function when called, it is opaque for the API
Supply custom logging function otherwise null
The singleton class OrtEnv contains the process-global ONNX Runtime environment.
It sets up logging, creates system wide thread-pools (if Thread Pool options are provided)
and other necessary things for OnnxRuntime to function.
Create or access OrtEnv by calling the Instance() method. Instance() can be called multiple times.
It would return the same instance.
CreateInstanceWithOptions() provides a way to create environment with options.
It must be called once before Instance() is called, otherwise it would not have effect.
If the environment is not explicitly created, it will be created as needed, e.g.,
when creating a SessionOptions instance.
The only __ctor__ for OrtEnv.
The actual logging callback to the native code
This is invoked only once when the first call refers _instance.Value.
To be called only from constructor
Instantiates (if not already done so) a new OrtEnv instance with the default logging level
and no other options. Otherwise returns the existing instance.
It returns the same instance on every call - `OrtEnv` is singleton
Returns a singleton instance of OrtEnv that represents native OrtEnv object
Provides a way to create an instance with options.
It throws if the instance already exists and the specified options
not have effect.
if the singleton has already been created
Provides visibility if singleton already been instantiated
Enable platform telemetry collection where applicable
(currently only official Windows ORT builds have telemetry collection capabilities)
Disable platform telemetry collection
Create and register an allocator to the OrtEnv instance
so as to enable sharing across all sessions using the OrtEnv instance
OrtMemoryInfo instance to be used for allocator creation
OrtArenaCfg instance that will be used to define the behavior of the arena based allocator
This function returns the onnxruntime version string
version string
Queries all the execution providers supported in the native onnxruntime shared library
an array of strings that represent execution provider names
Get/Set log level property of OrtEnv instance
Default LogLevel.Warning
env log level
Returns a handle to the native `OrtEnv` instance held by the singleton C# `OrtEnv` instance
Exception caching: May throw an exception on every call, if the `OrtEnv` constructor threw an exception
during lazy initialization
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Destroys native object
always returns true
Required because BitOperations are not available in NETSTANDARD 2.0.
There are more efficient ways with bit twiddling, but this one has clarity.
value
number of leading zeros. Useful to compute log2 as well.
Extracts single precision number bit representation as uint
so its bits can be manipulated.
This API is the reverse of UInt32BitsToSingle().
float value
Needed because BitConverter impl is not available until
later versions. This API is the reverse of SingleToUInt32Bits().
For the exact bit representation of float see IEEE 754 standard for single precision.
bit representation of float either obtained from
SingleToUInt32Bits or assembled using bitwise operators
Converts single precision bits representation which can be obtained using
SingleToUInt32Bits() or manually constructed according to IEEE 754 standard.
bits representation of a single precision number (float)
Converts bfloat16 ushort bits representation to single precision bits which then in turn can be
manipulated or converted to float using UInt32BitsToSingle()
ushort bits representation of bfloat16
Creates float NaN with the given sign and fp16 significand shifted << 54
true for negative
should be shifted 54 bits left before calling the function
so only 8 bits of significand remains
Creates float from sign, exponent and significand
true if negative
exponent
significand
This value type represents A Float16 value
it is blittable as defined in https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types
and as such, represented the same way in managed and native memories. This means that arrays of this type
do not have to be copied to be passed to native memory but simply pinned and read by native code. Thus,
one can create a Tensor on top of an array of these structures and feed it directly to Onnxruntime library.
Binary wise, it is the same as ushort[] (uint16_t in C++). However, we would like a separate type for type dispatching.
The implementation is derived from
https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/Half.cs,7895d5942d33f974
Float16 Epsilon value
Float16 Pi value
Float16 Positive Infinity value
Float16 Negative Infinity value
Float16 NaN
Float16 Zero value
Float16 One value
Float16 Negative Zero value
Float16 Lowest value
Float16 Max value
float16 representation bits
Ctor from ushort bits, no conversion is done
Compares values of two Float16
left hand side
right hand side
returns true if left is less than right according to IEEE
Compares values of two Float16
left hand side
right hand side
returns true if left is greater than right according to IEEE
Compares values of two Float16
left hand side
right hand side
returns true if left is less or equal than right according to IEEE
Compares values of two Float16
left hand side
right hand side
returns true if left is greater or equal than right according to IEEE
Compares values of two Float16 for binary equality.
If either of the values is NaN, this will return false.
left hand side
right hand side
true if values are equal according to IEEE
Compares values of two Float16 for binary inequality
true if values are not equal according to IEEE
Determines whether the specified value is finite (zero, subnormal, or normal).
Float16 instance.
true if the value is finite
Determines whether the specified value is infinite.
Float16 instance.
true if the value is infinite
Determines whether the specified value is NaN.
Float16 instance
true if the value is not a number
Determines whether the specified value is negative.
Float16 instance
true if the value is negative
Determines whether the specified value is negative infinity.
Float16 instance
true if the value is negative infinity
Determines whether the specified value is normal
true or false
Determines whether the specified value is positive infinity.
Float16 instance
Determines whether the specified value is subnormal.
Float16 instance
true if the value is subnormal
Compares this object to another object, returning an integer that indicates the relationship.
Object to compare to
A value less than zero if this is less than ,
zero if this is equal to , or a value greater than zero
if this is greater than .
Thrown when is not of type .
Compares this object to another object, returning an integer that indicates the relationship.
Object to compare to
A value less than zero if this is less than ,
zero if this is equal to ,
or a value greater than zero if this is greater than .
Returns a value indicating whether this instance and other Float16 represent the same value.
A Float16 object to compare to this instance.
true if other.value is equal to this instance; otherwise, false.
Returns a value indicating whether this instance and a specified System.Object
represent the same type and value.
An System.Object.
true if obj is Float16 and its value is equal to this instance; otherwise, false.
Returns the hash code for this instance.
A 32-bit signed integer hash code.
Returns a string representation of the current value.
Text representation of Float16
Explicit conversion
single precision value converted from Float16
Explicitly converts a value to its nearest representable half-precision floating-point value.
The value to convert.
converted to its nearest representable half-precision floating-point value.
Explicitly converts a half-precision floating-point value to its nearest representable value.
The value to convert.
converted to its nearest representable value.
Flips the sign. NaNs are not affected.
IEEE 754 specifies NaNs to be propagated
The function returns true if the value is either NaN or zero.
instance of Float16
true if NaN or zero.
This value type represents A BFloat16 value.
See https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus
for details.
it is blittable as defined in https://docs.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types
and as such, represented the same way in managed and native memories. This means that arrays of this type
do not have to be copied to be passed to native memory but simply pinnned and read by native code. Thus,
one can create a Tensor on top of an array of these structures and feed it directly to Onnxruntime library.
Binary wise, it is the same as ushort[] (uint16_t in C++). However, we would like a separate type for type dispatching.
BFloat16 Epsilon value
BFloat16 Pi value
BFloat16 Positive infinity value
BFloat16 Negative infinity value
BFloat16 NaN
BFloat16 Positive Zero
BFloat16 One
BFloat16 Negative Zero
BFloat16 Min value
BFloat16 Max value
bfloat16 representation bits
Constructor from ushort, no conversion takes place. The value
is assumed to be converted
bfloat16 representation bits
Compares two BFloat16 instances.
true if the left is less than right according to IEEE
Compares two BFloat16 instances.
true if the left is greater than right according to IEEE
Compares two BFloat16 instances.
true if the left is less or equal than right according to IEEE
Compares two BFloat16 instances.
true if the left is greater or equal than right according to IEEE
Compares values of two BFloat16 for binary equality.
If either of the values is NaN, this will return false.
left hand side
right hand side
result of value comparisons
Compares values of two BFloat16 for binary inequality
If either of the values is NaN it would return true.
result of value comparisons
Determines whether the specified value is finite (zero, subnormal, or normal).
BFloat16 instance.
true if the value is finite
Determines whether the specified value is infinite.
BFloat16 instance.
true if the value is infinite
Determines whether the specified value is NaN.
BFloat16 instance
true if the value is not a number
Determines whether the specified value is negative.
BFloat16 instance
true if the value is negative
Determines whether the specified value is negative infinity.
BFloat16 instance
true if the value is negative infinity
Determines whether the specified value is normal
true or false
Determines whether the specified value is positive infinity.
BFloat16 instance
Determines whether the specified value is subnormal.
BFloat16 instance
true if the value is subnormal
Compares this object to another object, returning an integer that indicates the relationship.
Object to compare to
A value less than zero if this is less than ,
zero if this is equal to , or a value greater than zero
if this is greater than .
Thrown when is not of type .
Compares this object to another object, returning an integer that indicates the relationship.
Object to compare to
A value less than zero if this is less than ,
zero if this is equal to ,
or a value greater than zero if this is greater than .
Returns a value indicating whether this instance and other BFloat16 represent the same value.
A BFloat16 object to compare to this instance.
true if other.value is equal to this instance; otherwise, false.
Returns a value indicating whether this instance and a specified System.Object
represent the same type and value.
An System.Object.
true if obj is BFloat16 its value is equal to this instance; otherwise, false.
Returns the hash code for this instance.
A 32-bit signed integer hash code.
Returns a string representation of the current value.
Text representation of BFloat16
Explicit conversion
single precision value converted from Float16
Explicitly converts a value to its nearest representable bfloat16 value.
The value to convert.
converted to its nearest representable half-precision floating-point value.
Explicitly converts a BFloat16 value to its nearest representable value.
The value to convert.
converted to its nearest representable value.
Flips the sign. NaNs are not affected.
IEEE 754 specifies NaNs to be propagated
The function returns true if the value is either NaN or zero.
instance of BFloat16
true if NaN or zero.
This class enables binding of inputs and/or outputs to pre-allocated
memory. This enables interesting scenarios. For example, if your input
already resides in some pre-allocated memory like GPU, you can bind
that piece of memory to an input name and shape and onnxruntime will use that as input.
Other traditional inputs can also be bound that already exists as Tensors.
Note, that this arrangement is designed to minimize data copies and to that effect
your memory allocations must match what is expected by the model, whether you run on
CPU or GPU. Data copy will still be made, if your pre-allocated memory location does not
match the one expected by the model. However, copies with OrtIoBindings are only done once,
at the time of the binding, not at run time. This means, that if your input data required a copy,
your further input modifications would not be seen by onnxruntime unless you rebind it, even if it is
the same buffer. If you require the scenario where data is copied, OrtIOBinding may not be the best match
for your use case. The fact that data copy is not made during runtime also has performance implications.
Making OrtValue first class citizen in ORT C# API practically obsoletes all of the existing overloads
because OrtValue can be created on top of the all other types of memory. No need to designate it as external
or Ort allocation or wrap it in FixedBufferOnnxValue. The latter does not support rebinding or memory other than
CPU anyway.
In fact, one can now create OrtValues over arbitrary pieces of memory, managed, native, stack and device(gpu)
and feed them to the model and achieve the same effect without using IOBinding class.
Use InferenceSession.CreateIoBinding()
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
This is the preferable and universal way to bind input to OrtValue.
This way you retain control over the original value, can modify the data
using OrtValue interfaces between the runs.
You can also create OrtValue on all kinds of memory, managed, native, stack and device(gpu).
input name
Bind a piece of pre-allocated native memory as a OrtValue Tensor with a given shape
to an input with a given name. The model will read the specified input from that memory
possibly avoiding the need to copy between devices. OrtMemoryAllocation continues to own
the chunk of native memory, and the allocation should be alive until the end of execution.
of the input
Tensor element type
native memory allocation
Bind externally (not from OrtAllocator) allocated memory as input.
The model will read the specified input from that memory
possibly avoiding the need to copy between devices. The user code continues to own
the chunk of externally allocated memory, and the allocation should be alive until the end of execution.
name
non ort allocated memory
Bind the input with the given name as an OrtValue Tensor allocated in pinned managed memory.
Instance of FixedBufferOnnxValue owns the memory and should be alive until the end of execution.
name of input
Blocks until device completes all preceding requested tasks.
Useful for memory synchronization.
This is the preferable and universal way to bind output via OrtValue.
This way you retain control over the original value, can modify the data
using OrtValue interfaces between the runs, rebind output to input if you
are feeding data circular.
output name
OrtValue to bind
Bind model output to an OrtValue as Tensor with a given type and shape. An instance of OrtMemoryAllocaiton
owns the memory and should be alive for the time of execution.
of the output
tensor element type
tensor shape
allocated memory
Bind externally (not from OrtAllocator) allocated memory as output.
The model will read the specified input from that memory
possibly avoiding the need to copy between devices. The user code continues to own
the chunk of externally allocated memory, and the allocation should be alive until the end of execution.
name
non ort allocated memory
Bind model output to a given instance of FixedBufferOnnxValue which owns the underlying
pinned managed memory and should be alive for the time of execution.
of the output
This function will bind model output with the given name to a device
specified by the memInfo.
output name
instance of memory info
Blocks until device completes all preceding requested tasks.
Useful for memory synchronization.
Bind allocation obtained from an Ort allocator
name
data type
tensor shape
ort allocation
whether this is input or output
Bind external allocation as input or output.
The allocation is owned by the user code.
name
non ort allocated memory
whether this is an input or output
Internal helper
Returns an array of output names in the same order they were bound
array of output names
This fetches bound outputs after running the model with RunWithBinding()
IDisposableReadOnlyCollection{OrtValue}
Clear all bound inputs and start anew
Clear all bound outputs
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtIoBidning
always returns true
Log severity levels
Must in sync with OrtLoggingLevel in onnxruntime_c_api.h
Represents Lora Adapter in memory
Creates an instance of OrtLoraAdapter from file.
The adapter file is memory mapped. If allocator parameter
is provided, then lora parameters are copied to the memory
allocated by the specified allocator.
path to the adapter file
optional allocator, can be null, must be a device allocator
New instance of LoraAdapter
Creates an instance of OrtLoraAdapter from an array of bytes. The API
makes a copy of the bytes internally.
array of bytes containing valid LoraAdapter format
optional device allocator or null
new instance of LoraAdapter
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtLoraAdapter
always returns true
This class allows to specify global thread pool options
when instantiating the ONNX Runtime environment for the first time.
A pointer to a underlying native instance of ThreadingOptions
Create an empty threading options.
Set global inter-op thread count.
Setting it to 0 will allow ORT to choose the number of threads used for parallelization of
multiple kernels. Setting it to 1 will cause the main thread to be used (i.e., no thread pools will be used).
This setting is only meaningful when the execution mode is set to Parallel.
The number of threads.
Sets the number of threads available for intra-op parallelism (i.e. within a single op).
Setting it to 0 will allow ORT to choose the number of threads, setting it to 1 will cause the main thread to be used (i.e., no thread pools will be used).
The number of threads.
Allows spinning of thread pools when their queues are empty in anticipation of imminent task arrival.
This call sets the value for both inter-op and intra-op thread pools.
If the CPU usage is very high then do not enable this.
If true allow the thread pools to spin.
When this is set it causes intra-op and inter-op thread pools to flush denormal values to zero.
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of ThreadingOptions
always returns true
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
This class retrieves Type Information for input/outputs of the model.
Represents OnnxValueType of the OrtTypeInfo
OnnxValueType
This property returns the tensor type and shape information from the OrtTypeInfo
iff this OrtTypeInfo represents a tensor.
Instance of OrtTensorTypeAndShapeInfo
Sequence type information from the OrtTypeInfo iff this OrtTypeInfo represents a sequence.
Instance of OrtSequenceTypeInfo
Represents MapTypeInfo from the OrtTypeInfo iff this OrtTypeInfo represents a map.
Instance of MapTypeInfo
Fetches OptionalTypeInfo from the OrtTypeInfo iff this OrtTypeInfo represents a optional type.
Instance of OptionalTypeInfo
This struct represents type and shape information for a tensor.
It may describe a tensor type that is a model input or output or
an information that can be extracted from a tensor in OrtValue.
Fetches tensor element data type
enum value for the data type
Returns true if this data element is a string
Fetches tensor element count based on the shape information.
number of typed tensor elements
Fetches shape dimension count (rank) for the tensor.
dim count
Tensor dimensions.
array of dims
Represents Sequence type information.
Returns type information for the element type of the sequence
OrtTypeInfo
Represents Map input/output information.
Maps are represented at run time by a tensor of primitive types
and values are represented either by Tensor/Sequence/Optional or another map.
Returns KeyType which is the data type of the keys tensor
OrtTypeInfo
Returns an instance of OrtTypeInfo describing the value type for the map
OrtTypeInfo
A type of data that OrtValue encapsulates.
Represents a disposable OrtValue.
This class exposes a native instance of OrtValue.
The class implements IDisposable and must
be disposed of, otherwise native resources will leak
and will eventually cause the application to slow down or crash.
If the OrtValue instance is constructed over a managed memory, and it is not
disposed properly, the pinned memory will continue to be pinned and interfere
with GC operation.
Constructor. The newly constructed OrtValue takes ownership of the native OrtValue instance
thrown when onnxValue type is not known
Constructor. The newly constructed OrtValue takes ownership of the native OrtValue instance
and disposes of it when the OrtValue instance is disposed. The instance will take ownership and will
dispose of compositeMembers instances.
This constructor can only throw if OnnxType is not specified.
native ortValue handle
must one of the valid types
For composite types this contains dependent ortValues such as members of a sequence
or keys/values for the map, that may have been created on top of the managed memory and must be disposed
with the new ortValue. This container will be taken the ownership of and the argument will be set to null.
throws when onnxValueType is not specified
Constructor to construct OrtValue over managed memory.
We pin the memory and unpin it at the disposal time.
The newly constructed OrtValue takes ownership of the native OrtValue instance
and disposes of it when the OrtValue instance is disposed.
Pointer to a native instance of OrtValue
memory handle to a pinned user supplied (usually managed) memory
It is disposed of (unpinned) when OrtValue is disposed.
Native handle to OrtValue for internal use.
Implement IOrtValueOwner interface
returns this
Fetches OrtValue type if it has one.
OnnxValueType
Returns true if OrtValue contains a tensor
true if tensor
Returns true if OrtValue contains a sparse tensor
true if sparse tensor
Valid for composite ML types like map, sequence.
Returns 2 for map (keys, values) and N for sequence, where N is the number of elements
in the sequence.
Element count
For non tensors return OrtValue element at the specified index.
For maps only indices 0 and 1 are valid. For sequences, [0..N) are valid.
See GetValueCount() to determine the valid range.
allocator to use
OrtValue disposable instance that points to the corresponding element of the composite type
Returns a ReadOnlySpan over tensor native buffer that
provides a read-only view.
Note, that the memory may be device allocated and, therefore, not accessible from the CPU.
To get memory descriptor use GetTensorMemoryInfo().
OrtValue must contain a non-string tensor.
The span is valid as long as the OrtValue instance is alive (not disposed).
ReadOnlySpan
Returns a ReadOnlyTensorSpan over tensor native buffer that
provides a read-only view.
Note, that the memory may be device allocated and, therefore, not accessible from the CPU.
To get memory descriptor use GetTensorMemoryInfo().
OrtValue must contain a non-string tensor.
The span is valid as long as the OrtValue instance is alive (not disposed).
ReadOnlySpan
Returns a Span over tensor native buffer.
This enables you to safely and efficiently modify the underlying
native buffer in a type-safe manner. This is useful for example in IOBinding scenarios
where you want to modify results of the inference and feed it back as input.
Note, that the memory may be device allocated.
To get memory descriptor use GetTensorMemoryInfo().
OrtValue must contain a non-string tensor.
The span is valid as long as the OrtValue instance is alive (not disposed).
Typed Span over the native buffer
Returns a TensorSpan over tensor native buffer.
Note, that the memory may be device allocated and, therefore, not accessible from the CPU.
To get memory descriptor use GetTensorMemoryInfo().
OrtValue must contain a non-string tensor.
The span is valid as long as the OrtValue instance is alive (not disposed).
ReadOnlySpan
Provides mutable raw native buffer access.
Span over the native buffer bytes
Provides mutable raw native buffer access.
TensorSpan over the native buffer bytes
Fetch string tensor element buffer pointer at the specified index,
convert/copy to UTF-16 char[] and return a ReadOnlyMemory{char} instance.
Obtain TensorTypeAndShape to get shape and element count.
flat string tensor element index
ReadOnlyMemory{char} backed by a managed char[]. Its lifespan is not
tied to the native buffer of OrtValue.
Fetch string tensor element buffer pointer at the specified index,
copy/convert UTF-8 into a UTF-16 string and return it.
Obtain TensorTypeAndShape to get shape and element count.
flat string tensor element index
UTF-16 string instance
Get a span over the native memory of the string tensor element.
The span is valid as long as the OrtValue is valid.
This is useful if you want to perform your own UTF-8 decoding or
you do not care about decoding.
Obtain TensorTypeAndShape to get shape and element count.
flat element index
ReadOnlySpan over UTF-8 bytes of the string tensor element
Convenience method to obtain all string tensor elements as a string array.
string[]
Creates and fetches Type information about the contained OnnxValue.
a disposable instance of OrtTypeInfo
Obtains Tensor And Type Information from the OrtValue iff it contains a tensor.
Valid only for OrtValues that contain a tensor.
A disposable instance of OrtTensorTypeAndShapeInfo
Returns OrtMemoryInfo iff this OrtValue contains a tensor or a sparse tensor.
OrtMemoryInfo that describes the underlying memory allocation
Factory method to construct an OrtValue of Tensor type on top of pre-allocated memory.
This can be a piece of arbitrary memory that may be allocated by OrtAllocator (possibly on a device),
a chunk of managed memory (must be pinned for the duration of OrtValue lifetime) or a memory that is allocated
natively allocated using Marshal.AllocHGlobal(), stackalloc or other means (may be on a device).
The resulting OrtValue does not own the underlying memory buffer and will not attempt to
deallocate it. The caller must make sure that the memory remains valid for the duration of OrtValue lifetime.
Memory Info. For managed memory its default is cpu.
For other kinds of memory, one must construct as appropriate.
DataType for the Tensor
shape of the tensor to create. The size required by the shape
must be less of equal of the memory.Length
Pointer to a raw memory buffer which may reside on a device
Buffer length in bytes
A disposable instance of OrtValue
This is a factory method that creates an OrtValue of Tensor type on top of Memory memory.
The API pins the memory for the duration of the OrtValue lifetime.
It is unpinned at disposal time.
T must be one of the supported types
Memory information that describes memory location
contiguous region of memory
shape of the tensor to create. The size required by the shape
must be less of equal of the memory.Length
A disposable OrtValue instance
This is a factory method that creates an OrtValue of Tensor type on top managed data array.
The API pins the memory for the duration of the OrtValue lifetime.
It is unpinned at disposal time.
managed data buffer
shape that describes the buffer
A disposable OrtValue instance
This is a factory method creates a native Onnxruntime OrtValue containing a tensor on top of the existing tensor managed memory.
The method will attempt to pin managed memory so no copying occurs when data is passed down
to native code.
Tensor object
discovered tensor element type
And instance of OrtValue constructed on top of the object
The factory API creates an OrtValue with memory allocated using the given allocator
according to the specified shape and element type. The memory will be released when OrtValue
is disposed. Use GetTensorMutableDataAsSpan<T>() API to fill in the data.
A disposable OrtValue
This is a factory method creates a native Onnxruntime OrtValue containing a tensor.
The method will attempt to pin managed memory so no copying occurs when data is passed down
to native code.
Tensor object
discovered tensor element type
And instance of OrtValue constructed on top of the object
Creates an OrtValue that contains a string tensor of specified shape, and
containing empty strings. String tensors are always on CPU.
Use StringTensorSetElementAt to assign individual elements values.
disposable OrtValue
tensor shape
Converts the string argument represented by ReadOnlySpan to UTF-8,
allocates space in the native tensor and copies it into the native tensor memory.
Typically, this is used to populate a new empty string tensor element.
The number of elements is according to the shape supplied to CreateTensorWithEmptyStrings().
However, this API can also be used to overwrite any existing element within the string tensor.
In general, to obtain the number of elements for any tensor, use GetTensorTypeAndShape() which
would return a disposable instance of TensorTypeAndShapeInfo.
Then call GetElementCount() or GetShape().
ReadOnlySpan over chars
index of the string element within the tensor
must be within bounds of [0, N)
Converts the string argument represented by ReadOnlyMemory to UTF-8,
allocates space in the native tensor and copies it into the native tensor memory.
Typically, this is used to populate a new empty string tensor element.
The number of elements is according to the shape supplied to CreateTensorWithEmptyStrings().
However, this API can also be used to overwrite any existing element within the string tensor.
In general, to obtain the number of elements for any tensor, use GetTensorTypeAndShape() which
would return a disposable instance of TensorTypeAndShapeInfo.
Then call GetElementCount() or GetShape().
ReadOnlyMemory instance over an array of chars
index of the string element within the tensor
must be within bounds of [0, N)
This API resizes String Tensor element to the requested amount of bytes (UTF-8)
and copies the bytes from the supplied ReadOnlySpan into the native tensor memory (resized buffer).
The API is useful for quick loading of utf8 data into the native tensor memory.
read only span of bytes
flat index of the element in the string tensor
Creates an OrtValue that contains a string tensor.
String tensors are always allocated on CPU.
String data will be converted to UTF-8 and copied to native memory.
Note, this is different from creating an OrtValue from other primitive data types
where memory is pinned (if necessary) and the OrtValue points to that chunk of memory.
Tensor{string}
A disposable OrtValue instance
Creates a sequence of OrtValues from a collection of OrtValues.
All OrtValues in the collection must be of the same Onnx type.
I.e. (Tensor, SparseTensor, Map, Sequence, etc.)
The ortValues that are passed as argument are taken possession of by the newly
created OrtValue. The caller should not dispose them, unless this call fails.
The ortValues would be empty on successful return.
a collection of OrtValues. On success the ortValues contained in the list
are taken ownership of and the list is cleared.
A disposable instance of OrtValues
Creates a sequence from the values in compositeMembers
The argument is taken possession of and is nullified on successful return.
sequence ortValues
OrtValue instance representing a Sequence
A delegate type that is expected to process each OrtValue in a sequence.
OrtValue that holds sequence element
ordinal of the value
Feeds each OrtValue in a sequence to the visitor delegate.
This helps users to avoid dealing each value life-span
visitor delegate
allocator to use for intermediate ort values
Creates a map OrtValue with keys and values.
On a high level the Onnxruntime representation of the map always consists of two
OrtValues, keys and values.
According to ONNX standard map keys can be unmanaged types only (or strings).
Those keys are contained in a single tensor within OrtValue keys.
Map values, on the other hand, can be composite types. The values parameter
can either contain a single tensor with unmanaged map values with the same number of
elements as the keys, or it can be a sequence of OrtValues,
each of those can be a composite type (tensor, sequence, map). If it is a sequence,
then the number of elements must match the number of elements in keys.
Keys and values must be in the same order.
ORT supports only a subset of types for keys and values, however, this API does not
restrict it.
The ortValues that are passed as argument are taken possession of by the newly
created OrtValue. The caller should not dispose them, unless this call fails.
Keys and values arguments will be set to null on success.
Contains keys
Contains values
A disposable OrtValue
This API helps to quickly creates a map OrtValue with unmanaged (primitive) keys and values specified as arrays.
This helps the user not to create OrtValues for keys and values separately and deal only with the final result.
The map would consist of two tensors, one for keys and one for values.
The OrtValues would be created on top of the managed memory arrays and use it directly.
The number of elements in keys and values must be the same and they must be in order.
The types must be unmanaged.
keys type
values type
array of keys of K type
array of values of V type
OrtValue instance
Creates a map OrtValue with string keys and non-string values.
This helps the user not to create OrtValues for keys and values separately.
The number of elements in keys and values must be the same and they must be in order.
The map would consist of two tensors, one for keys and one for values.
string keys would be converted to UTF-8 encoding and copied to an allocated native memory.
The OrtValue for values would be created on top of the managed memory using it directly.
The values type must be unmanaged.
Collection of strings
OrtValue instance
Creates a map OrtValue with non-string keys and string values.
This helps the user not to create OrtValues for keys and values separately.
The number of elements in keys and values must be the same and they must be in order.
The OrtValue for keys would be created on top of the managed memory using it directly.
string values would be converted to UTF-8 encoding and copied to an allocated native memory.
unmanaged type of keys
collection of string values
Instance of OrtValue
A public delegate that will be invoked once with map keys and values.
The delegate helps not to deal with the lifespan of intermediate OrtValues.
Typically, when one uses GetValue() API, it creates a copy of OrtValue
that points to the same buffer as keys or values. This API helps to deal with those
temporary instances and avoid leaks.
According to ONNX standard map keys can be unmanaged types only (or strings).
Those keys are contained in a single tensor within OrtValue keys. So you can query those
directly from keys argument.
Map values, on the other hand, can be composite types. The values parameter
can either contain a single tensor with unmanaged map values with the same number of
elements as the keys, or it can be a sequence of OrtValues,
each of those can be a composite type (tensor, sequence, map). If it is a sequence,
then the number of elements must match the number of elements in keys.
Depending on the structure of the values, one will either directly query a single tensor
from values, or will have to iterate over the sequence of OrtValues and visit each of those
resulting in a recursive visitation.
This would always represent a tensor
Can be any of the Onnx types, but they would all reduce to tensors eventually
This API helps the user to process a map OrtValue without
having to deal with the lifespan of intermediate OrtValues.
each API value is fed to the vistor functor.
visitor function
Allocator to use for intermediate values
IDisposable implementation
IDisposable implementation
true if invoked from Dispose() method
Provides access from the underlying object that owns disposable OrtValue
The returned value does not own the actual memory and does nothing on Dispose()
This class is used in conjunction with DisposableNamedOnnxValue to
own native collection OrtValue and dispose of it along with any DisposableNamedOnnxValues
_Ctor. Takes ownership of ortValue and sets it to null on success.
becomes null on success
A collection of disposables that support composed types.
We stick them here and dispose them when this instance is disposed.
Returns OrtValue that is owned by this instance
This helper class owns the underlying OrtValue that is assumed to be a Tensor,
it does not support any other ortValues and caches Tensor properties.
It is easy to expose as a Tensor{T} as DenseTensor can take Memory Mapping from
this.
This class is disposable because of the MemoryManager inheritance. Because this class
always backs exactly only one DenseTensor instance, it does
not implement ref-counting for Pin/Unpin.
Constructs an instance and takes ownership of ortValue on success
ortValue that is a Tensor. It becomes null on successful return.
Returns OrtValue that is owned by this instance
Returns Span that is a view into the underlying native Tensor memory
SpanT
Satisfy MemoryManager abstract implementation.
required for override
This class holds pre-packed weights of shared initializers to be shared across sessions using these initializers
and thereby provide memory savings by sharing the same pre-packed versions of these shared initializers
Constructs an empty PrePackedWeightsContainer
Internal accessor to call native methods
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Overrides SafeHandle.ReleaseHandle() to deallocate
a chunk of memory using the specified allocator.
always returns true
Holds the options for configuring a TensorRT Execution Provider instance
Constructs an empty OrtTensorRTProviderOptions instance
Get TensorRT EP provider options
return C# UTF-16 encoded string
Updates the configuration knobs of OrtTensorRTProviderOptions that will eventually be used to configure a TensorRT EP
Please refer to the following on different key/value pairs to configure a TensorRT EP and their meaning:
https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html
key/value pairs used to configure a TensorRT Execution Provider
Get device id of TensorRT EP.
device id
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtTensorRTProviderOptions
always returns true
Holds the options for configuring a CUDA Execution Provider instance
Constructs an empty OrtCUDAroviderOptions instance
Get CUDA EP provider options
return C# UTF-16 encoded string
Updates the configuration knobs of OrtCUDAProviderOptions that will eventually be used to configure a CUDA EP
Please refer to the following on different key/value pairs to configure a CUDA EP and their meaning:
https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html
key/value pairs used to configure a CUDA Execution Provider
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtCUDAProviderOptions
always returns true
Holds the options for configuring a ROCm Execution Provider instance
Constructs an empty OrtROCMroviderOptions instance
Get ROCm EP provider options
return C# UTF-16 encoded string
Updates the configuration knobs of OrtROCMProviderOptions that will eventually be used to configure a ROCm EP
Please refer to the following on different key/value pairs to configure a ROCm EP and their meaning:
https://onnxruntime.ai/docs/execution-providers/ROCm-ExecutionProvider.html
key/value pairs used to configure a ROCm Execution Provider
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of OrtROCMProviderOptions
always returns true
This helper class contains methods to handle values of provider options
Parse from string and save to dictionary
C# string
Dictionary instance to store the parsing result of s
CoreML flags for use with SessionOptions.
See https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/providers/coreml/coreml_provider_factory.h
NNAPI flags for use with SessionOptions.
See https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/providers/nnapi/nnapi_provider_factory.h
Sets various runtime options.
Default __ctor. Creates default RuntimeOptions
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Log Severity Level for the session logs. Default = ORT_LOGGING_LEVEL_WARNING
Log Verbosity Level for the session logs. Default = 0. Valid values are >=0.
This takes into effect only when the LogSeverityLevel is set to ORT_LOGGING_LEVEL_VERBOSE.
Log tag to be used during the run. default = ""
Sets a flag to terminate all Run() calls that are currently using this RunOptions object
Default = false
terminate flag value
Set a single run configuration entry as a pair of strings
If a configuration with same key exists, this will overwrite the configuration with the given configValue.
config key name
config key value
Appends the specified lora adapter to the list of active lora adapters
for this RunOptions instance. All run calls with this instant will
make use of the activated Lora Adapters. An adapter is considered active
if it is added to RunOptions that are used during Run() calls.
Lora adapter instance
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of RunOptions
always returns true
Graph optimization level to use with SessionOptions
[https://github.com/microsoft/onnxruntime/blob/main/docs/ONNX_Runtime_Graph_Optimizations.md]
Controls whether you want to execute operators in the graph sequentially or in parallel.
Usually when the model has many branches, setting this option to ExecutionMode.ORT_PARALLEL
will give you better performance.
See [ONNX_Runtime_Perf_Tuning.md] for more details.
Holds the options for creating an InferenceSession
It forces the instantiation of the OrtEnv singleton.
Constructs an empty SessionOptions
A helper method to construct a SessionOptions object for CUDA execution.
Use only if CUDA is installed and you have the onnxruntime package specific to this Execution Provider.
A SessionsOptions() object configured for execution on deviceId
A helper method to construct a SessionOptions object for CUDA execution provider.
Use only if CUDA is installed and you have the onnxruntime package specific to this Execution Provider.
CUDA EP provider options
A SessionsOptions() object configured for execution on provider options
A helper method to construct a SessionOptions object for TensorRT execution.
Use only if CUDA/TensorRT are installed and you have the onnxruntime package specific to this Execution Provider.
A SessionsOptions() object configured for execution on deviceId
A helper method to construct a SessionOptions object for TensorRT execution provider.
Use only if CUDA/TensorRT are installed and you have the onnxruntime package specific to this Execution Provider.
TensorRT EP provider options
A SessionsOptions() object configured for execution on provider options
A helper method to construct a SessionOptions object for ROCM execution.
Use only if ROCM is installed and you have the onnxruntime package specific to this Execution Provider.
Device Id
A SessionsOptions() object configured for execution on deviceId
A helper method to construct a SessionOptions object for ROCm execution provider.
Use only if ROCm is installed and you have the onnxruntime package specific to this Execution Provider.
ROCm EP provider options
A SessionsOptions() object configured for execution on provider options
Appends CPU EP to a list of available execution providers for the session.
1 - use arena, 0 - do not use arena
Use only if you have the onnxruntime package specific to this Execution Provider.
1 - use allocation arena, 0 - otherwise
Use only if you have the onnxruntime package specific to this Execution Provider.
integer device ID
Append a CUDA EP instance (based on specified configuration) to the SessionOptions instance.
Use only if you have the onnxruntime package specific to this Execution Provider.
CUDA EP provider options
Use only if you have the onnxruntime package specific to this Execution Provider.
device identification
Use only if you have the onnxruntime package specific to this Execution Provider.
device identification, default empty string
Use only if you have the onnxruntime package specific to this Execution Provider.
device identification
Append a TensorRT EP instance (based on specified configuration) to the SessionOptions instance.
Use only if you have the onnxruntime package specific to this Execution Provider.
TensorRT EP provider options
Use only if you have the onnxruntime package specific to this Execution Provider.
Device Id
Append a ROCm EP instance (based on specified configuration) to the SessionOptions instance.
Use only if you have the onnxruntime package specific to this Execution Provider.
ROCm EP provider options
Use only if you have the onnxruntime package specific to this Execution Provider.
device identification
Use only if you have the onnxruntime package specific to this Execution Provider.
NNAPI specific flag mask
Use only if you have the onnxruntime package specific to this Execution Provider.
CoreML specific flags
Append QNN, SNPE or XNNPACK execution provider
Execution provider to add. 'QNN', 'SNPE' 'XNNPACK', 'CoreML and 'AZURE are currently supported.
Optional key/value pairs to specify execution provider options.
Loads a DLL named 'libraryPath' and looks for this entry point:
OrtStatus* RegisterCustomOps(OrtSessionOptions* options, const OrtApiBase* api);
It then passes in the provided session options to this function along with the api base.
Prior to v1.15 this leaked the library handle and RegisterCustomOpLibraryV2
was added to resolve that.
From v1.15 on ONNX Runtime will manage the lifetime of the handle.
path to the custom op library
Loads a DLL named 'libraryPath' and looks for this entry point:
OrtStatus* RegisterCustomOps(OrtSessionOptions* options, const OrtApiBase* api);
It then passes in the provided session options to this function along with the api base.
The handle to the loaded library is returned in 'libraryHandle'.
It can be unloaded by the caller after all sessions using the passed in
session options are destroyed, or if an error occurs and it is non null.
Hint: .NET Core 3.1 has a 'NativeLibrary' class that can be used to free the library handle
Custom op library path
out parameter, library handle
Register the custom operators from the Microsoft.ML.OnnxRuntime.Extensions NuGet package.
A reference to Microsoft.ML.OnnxRuntime.Extensions must be manually added to your project.
Throws if the extensions library is not found.
Add a pre-allocated initializer to a session. If a model contains an initializer with a name
that is same as the name passed to this API call, ORT will use this initializer instance
instead of deserializing one from the model file. This is useful when you want to share
the same initializer across sessions.
name of the initializer
OrtValue containing the initializer. Lifetime of 'val' and the underlying initializer buffer must be
managed by the user (created using the CreateTensorWithDataAsOrtValue API) and it must outlive the session object
Set a single session configuration entry as a pair of strings
If a configuration with same key exists, this will overwrite the configuration with the given configValue
config key name
config key value
Override symbolic dimensions (by specific denotation strings) with actual values if known at session initialization time to enable
optimizations that can take advantage of fixed values (such as memory planning, etc)
denotation name
denotation value
Override symbolic dimensions (by specific name strings) with actual values if known at session initialization time to enable
optimizations that can take advantage of fixed values (such as memory planning, etc)
dimension name
dimension value
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Enables the use of the memory allocation patterns in the first Run() call for subsequent runs. Default = true.
returns enableMemoryPattern flag value
Path prefix to use for output of profiling data
Enables profiling of InferenceSession.Run() calls. Default is false
returns _enableProfiling flag value
Set filepath to save optimized model after graph level transformations. Default is empty, which implies saving is disabled.
returns _optimizedModelFilePath flag value
Enables Arena allocator for the CPU memory allocations. Default is true.
returns _enableCpuMemArena flag value
Disables the per session threads. Default is true.
This makes all sessions in the process use a global TP.
Log Id to be used for the session. Default is empty string.
returns _logId value
Log Severity Level for the session logs. Default = ORT_LOGGING_LEVEL_WARNING
returns _logSeverityLevel value
Log Verbosity Level for the session logs. Default = 0. Valid values are >=0.
This takes into effect only when the LogSeverityLevel is set to ORT_LOGGING_LEVEL_VERBOSE.
returns _logVerbosityLevel value
Sets the number of threads used to parallelize the execution within nodes
A value of 0 means ORT will pick a default
returns _intraOpNumThreads value
Sets the number of threads used to parallelize the execution of the graph (across nodes)
If sequential execution is enabled this value is ignored
A value of 0 means ORT will pick a default
returns _interOpNumThreads value
Sets the graph optimization level for the session. Default is set to ORT_ENABLE_ALL.
returns _graphOptimizationLevel value
Sets the execution mode for the session. Default is set to ORT_SEQUENTIAL.
See [ONNX_Runtime_Perf_Tuning.md] for more details.
returns _executionMode value
Sets the load cancellation flag for the session. Default is set to false.
Provides an opportunity for the user to cancel model loading.
true to request cancellation, false to proceed
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of SessionOptions
always returns true
Helper to allow the creation/addition of session options based on pre-defined named entries.
Register the default handler. This is used when a configuration name is not provided.
Handler that applies the default settings to a SessionOptions instance.
Register a named handler.
Configuration name.
Handler that applies the settings for the configuration to a SessionOptions instance.
Create a SessionOptions instance with configuration applied.
Configuration to use.
If not provided, the default set of session options will be applied if useDefaultAsFallback is true.
If configuration is not provided or not found, use the default session options.
SessionOptions with configuration applied.
Reset by removing all registered handlers.
Apply a configuration to a SessionOptions instance.
SessionOptions to apply configuration to.
Configuration to use.
Use the default configuration if 'configuration' is not provided or not found.
Updated SessionOptions instance.
This class contains utilities for useful calculations with shape.
Returns a number of elements in the tensor from the given shape
size
Gets the set of strides that can be used to calculate the offset of n-dimensions in a 1-dimensional layout
an array of strides
Calculates the 1-d index for n-d indices in layout specified by strides.
pre-calculated strides
Indices. Must have the same length as strides
A 1-d index into the tensor buffer
A static class that houses static DenseTensor{T} extension methods
Creates a copy of this single-dimensional array as a DenseTensor<T>
Type contained in the array to copy to the DenseTensor<T>.
The array to create a DenseTensor<T> from.
A 1-dimensional DenseTensor<T> with the same length and content as .
Creates a copy of this two-dimensional array as a DenseTensor<T>
Type contained in the array to copy to the DenseTensor<T>.
The array to create a DenseTensor<T> from.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension is most minor (closest together): row-major. True to indicate that the last dimension is most major (farthest apart) and the first dimension is most minor (closest together): column-major.
A 2-dimensional DenseTensor<T> with the same dimensions and content as .
Creates a copy of this three-dimensional array as a DenseTensor<T>
Type contained in the array to copy to the DenseTensor<T>.
The array to create a DenseTensor<T> from.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension is most minor (closest together): akin to row-major in a rank-2 tensor. True to indicate that the last dimension is most major (farthest apart) and the first dimension is most minor (closest together): akin to column-major in a rank-2 tensor.
A 3-dimensional DenseTensor<T> with the same dimensions and content as .
Creates a copy of this four-dimensional array as a DenseTensor<T>
Type contained in the array to copy to the DenseTensor<T>.
The array to create a DenseTensor<T> from.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension is most minor (closest together): akin to row-major in a rank-2 tensor. True to indicate that the last dimension is most major (farthest apart) and the first dimension is most minor (closest together): akin to column-major in a rank-2 tensor.
A 4-dimensional DenseTensor<T> with the same dimensions and content as .
Creates a copy of this n-dimensional array as a DenseTensor<T>
Type contained in the array to copy to the DenseTensor<T>.
The array to create a DenseTensor<T> from.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension is most minor (closest together): akin to row-major in a rank-2 tensor. True to indicate that the last dimension is most major (farthest apart) and the first dimension is most minor (closest together): akin to column-major in a rank-2 tensor.
A n-dimensional DenseTensor<T> with the same dimensions and content as .
Gets the set of strides that can be used to calculate the offset of n-dimensions in a 1-dimensional layout
Calculates the 1-d index for n-d indices in layout specified by strides.
Calculates the n-d indices from the 1-d index in a layout specified by strides
Calculates the n-d indices from the 1-d index in a layout specificed by strides
Takes an 1-d index over n-d sourceStrides and recalculates it assuming same n-d coordinates over a different n-d strides
Represents a multi-dimensional collection of objects of type T that can be accessed by indices.
DenseTensor stores values in a contiguous sequential block of memory where all values are represented.
Type contained within the Tensor. Typically a value type such as int, double, float, etc.
Initializes a rank-1 Tensor using the specified .
Size of the 1-dimensional tensor
Initializes a rank-n Tensor using the dimensions specified in .
An span of integers that represent the size of each dimension of the DenseTensor to create.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension
is most minor (closest together): akin to row-major in a rank-2 tensor.
True to indicate that the last dimension is most major (farthest apart) and the first dimension is most
minor (closest together): akin to column-major in a rank-2 tensor.
Constructs a new DenseTensor of the specified dimensions, wrapping existing backing memory for the contents.
An span of integers that represent the size of each dimension of the DenseTensor to create.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension
is most minor (closest together): akin to row-major in a rank-2 tensor.
True to indicate that the last dimension is most major (farthest apart) and the first dimension is most
minor (closest together): akin to column-major in a rank-2 tensor.
Memory storing backing values of this tensor.
Gets the value at the specified index, where index is a linearized version of n-dimension indices
using strides. For a scalar, use index = 0
An integer index computed as a dot-product of indices.
The value at the specified position in this Tensor.
Sets the value at the specified index, where index is a linearized version of n-dimension indices
using strides. For a scalar, use index = 0
An integer index computed as a dot-product of indices.
The new value to set at the specified position in this Tensor.
Overrides Tensor.CopyTo(). Copies the content of the Tensor
to the specified array starting with arrayIndex
destination array
start index
Determines the index of a specific item in the Tensor<T>.
Object to locate
The index of item if found in the tensor; otherwise, -1
Creates a shallow copy of this tensor, with new backing storage.
A shallow copy of this tensor.
Creates a new Tensor of a different type with the specified dimensions and the same layout as this tensor
with elements initialized to their default value.
Type contained in the returned Tensor.
An span of integers that represent the size of each dimension of the DenseTensor to create.
A new tensor with the same layout as this tensor but different type and dimensions.
Reshapes the current tensor to new dimensions, using the same backing storage.
An span of integers that represent the size of each dimension of the DenseTensor to create.
A new tensor that reinterprets backing Buffer of this tensor with different dimensions.
Supported Tensor DataType
Helps typecasting. Holds Tensor element type traits.
TensorElementType enum
type enum value
Size of the stored primitive type in bytes
size in bytes
Is the type is a string
true if Tensor element type is a string
Ctor
TensorElementType value
size fo the type in bytes
Holds TensorElement traits
Tensor element type
System.Type
Size of the stored primitive type in bytes
size in bytes
Is the type is a string
true if Tensor element type is a string
Ctor
Tensor element type
typesize
This class is a base for all Tensors. It hosts maps with type traits.
Constructs TensorBae
primitive type the deriving class is using
Query TensorTypeInfo by one of the supported types
TensorTypeInfo or null if not supported
Query TensorElementTypeInfo by enum
type enum
instance of TensorElementTypeInfo or null if not found
Query TensorTypeInfo using this Tensor type
Various methods for creating and manipulating Tensor<T>
Creates an identity tensor of the specified size. An identity tensor is a two dimensional tensor with 1s in the diagonal.
type contained within the Tensor. Typically a value type such as int, double, float, etc.
Width and height of the identity tensor to create.
a by with 1s along the diagonal and zeros elsewhere.
Creates an identity tensor of the specified size and layout (row vs column major). An identity tensor is a two dimensional tensor with 1s in the diagonal.
type contained within the Tensor. Typically a value type such as int, double, float, etc.
Width and height of the identity tensor to create.
>False to indicate that the first dimension is most minor (closest) and the last dimension is most major (farthest): row-major. True to indicate that the last dimension is most minor (closest together) and the first dimension is most major (farthest apart): column-major.
a by with 1s along the diagonal and zeros elsewhere.
Creates an identity tensor of the specified size and layout (row vs column major) using the specified one value. An identity tensor is a two dimensional tensor with 1s in the diagonal. This may be used in case T is a type that doesn't have a known 1 value.
type contained within the Tensor. Typically a value type such as int, double, float, etc.
Width and height of the identity tensor to create.
>False to indicate that the first dimension is most minor (closest) and the last dimension is most major (farthest): row-major. True to indicate that the last dimension is most minor (closest together) and the first dimension is most major (farthest apart): column-major.
Value of that is used along the diagonal.
a by with 1s along the diagonal and zeros elsewhere.
Creates a n+1-rank tensor using the specified n-rank diagonal. Values not on the diagonal will be filled with zeros.
type contained within the Tensor. Typically a value type such as int, double, float, etc.
Tensor representing the diagonal to build the new tensor from.
A new tensor of the same layout and order as of one higher rank, with the values of along the diagonal and zeros elsewhere.
Creates a n+1-dimension tensor using the specified n-dimension diagonal at the specified offset
from the center. Values not on the diagonal will be filled with zeros.
type contained within the Tensor. Typically a value type such as int, double, float, etc.
Tensor representing the diagonal to build the new tensor from.
Offset of diagonal to set in returned tensor. 0 for the main diagonal,
less than zero for diagonals below, greater than zero from diagonals above.
A new tensor of the same layout and order as of one higher rank,
with the values of along the specified diagonal and zeros elsewhere.
Represents a multi-dimensional collection of objects of type T that can be accessed by indices.
type contained within the Tensor. Typically a value type such as int, double, float, etc.
Initialize a 1-dimensional tensor of the specified length
Size of the 1-dimensional tensor
Initialize an n-dimensional tensor with the specified dimensions and layout.
ReverseStride=true gives a stride of 1-element width to the first dimension (0).
ReverseStride=false gives a stride of 1-element width to the last dimension (n-1).
An span of integers that represent the size of each dimension of the Tensor to create.
False (default) to indicate that the first dimension is most major (farthest apart) and the last dimension
is most minor (closest together): akin to row-major in a rank-2 tensor.
True to indicate that the last dimension is most major (farthest apart) and the first dimension is most
minor (closest together): akin to column-major in a rank-2 tensor.
If you pass `null` for dimensions it will implicitly convert to an empty ReadOnlySpan, which is
equivalent to the dimensions for a scalar value.
Initializes tensor with same dimensions as array, content of array is ignored.
ReverseStride=true gives a stride of 1-element width to the first dimension (0).
ReverseStride=false gives a stride of 1-element width to the last dimension (n-1).
Array from which to derive dimensions.
False (default) to indicate that the first dimension is most major (farthest apart) and the
last dimension is most minor (closest together): akin to row-major in a rank-2 tensor.
True to indicate that the last dimension is most major (farthest apart) and the first dimension
is most minor (closest together): akin to column-major in a rank-2 tensor.
Total length of the Tensor.
Rank of the tensor: number of dimensions.
True if strides are reversed (AKA Column-major)
Returns a readonly view of the dimensions of this tensor.
Returns a readonly view of the strides of this tensor.
Sets all elements in Tensor to .
Value to fill
Creates a shallow copy of this tensor, with new backing storage.
A shallow copy of this tensor.
Creates a new Tensor with the same layout and dimensions as this tensor with elements initialized to their default value.
A new Tensor with the same layout and dimensions as this tensor with elements initialized to their default value.
Creates a new Tensor with the specified dimensions and the same layout as this tensor with elements initialized to their default value.
An span of integers that represent the size of each dimension of the DenseTensor to create.
A new Tensor with the same layout as this tensor and specified with elements initialized to their default value.
Creates a new Tensor of a different type with the same layout and size as this tensor with elements initialized to their default value.
Type contained within the new Tensor. Typically a value type such as int, double, float, etc.
A new Tensor with the same layout and dimensions as this tensor with elements of type initialized to their default value.
Creates a new Tensor of a different type with the specified dimensions and the same layout as this tensor with elements initialized to their default value.
Type contained within the new Tensor. Typically a value type such as int, double, float, etc.
An span of integers that represent the size of each dimension of the DenseTensor to create.
A new Tensor with the same layout as this tensor of specified with elements of type initialized to their default value.
Gets the n-1 dimension diagonal from the n dimension tensor.
An n-1 dimension tensor with the values from the main diagonal of this tensor.
Gets the n-1 dimension diagonal from the n dimension tensor at the specified offset from center.
Offset of diagonal to set in returned tensor. 0 for the main diagonal, less than zero for diagonals below, greater than zero from diagonals above.
An n-1 dimension tensor with the values from the specified diagonal of this tensor.
Gets a tensor representing the elements below and including the diagonal, with the rest of the elements zero-ed.
A tensor with the values from this tensor at and below the main diagonal and zeros elsewhere.
Gets a tensor representing the elements below and including the specified diagonal, with the rest of the elements zero-ed.
Offset of diagonal to set in returned tensor. 0 for the main diagonal, less than zero for diagonals below, greater than zero from diagonals above.
A tensor with the values from this tensor at and below the specified diagonal and zeros elsewhere.
Gets a tensor representing the elements above and including the diagonal, with the rest of the elements zero-ed.
A tensor with the values from this tensor at and above the main diagonal and zeros elsewhere.
Gets a tensor representing the elements above and including the specified diagonal, with the rest of the elements zero-ed.
Offset of diagonal to set in returned tensor. 0 for the main diagonal, less than zero for diagonals below, greater than zero from diagonals above.
A tensor with the values from this tensor at and above the specified diagonal and zeros elsewhere.
Implementation method for GetTriangle, GetLowerTriangle, GetUpperTriangle
Offset of diagonal to set in returned tensor.
true for upper triangular and false otherwise
Reshapes the current tensor to new dimensions, using the same backing storage if possible.
An span of integers that represent the size of each dimension of the Tensor to create.
A new tensor that reinterprets this tensor with different dimensions.
Obtains the value at the specified indices
A one-dimensional array of integers that represent the indices specifying the position of the element to get.
The value at the specified position in this Tensor.
Obtains the value at the specified indices
A span integers that represent the indices specifying the position of the element to get.
The value at the specified position in this Tensor.
Gets the value at the specied index, where index is a linearized version of n-dimension indices using strides.
An integer index computed as a dot-product of indices.
The value at the specified position in this Tensor.
Sets the value at the specied index, where index is a linearized version of n-dimension indices using strides.
An integer index computed as a dot-product of indices.
The new value to set at the specified position in this Tensor.
Performs a value comparison of the content and shape of two tensors. Two tensors are equal if they have the same shape and same value at every set of indices. If not equal a tensor is greater or less than another tensor based on the first non-equal element when enumerating in linear order.
Performs a value equality comparison of the content of two tensors. Two tensors are equal if they have the same shape and same value at every set of indices.
Always fixed size Tensor
always true
Tensor is not readonly
always false
Determines whether an element is in the Tensor<T>.
The object to locate in the Tensor<T>. The value can be null for reference types.
true if item is found in the Tensor<T>; otherwise, false.
Copies the elements of the Tensor<T> to an Array, starting at a particular Array index.
The one-dimensional Array that is the destination of the elements copied from Tensor<T>. The Array must have zero-based indexing.
The zero-based index in array at which copying begins.
Determines the index of a specific item in the Tensor<T>.
The object to locate in the Tensor<T>.
The index of item if found in the tensor; otherwise, -1.
Creates a copy of this tensor as a DenseTensor<T>. If this tensor is already a DenseTensor<T> calling this method is equivalent to calling Clone().
Get a string representation of Tensor
Holds the state of the training session.
This class holds the entire training session state that includes model parameters, their gradients,
optimizer parameters, and user properties. The TrainingSession leverages the CheckpointState
by accessing and updating the contained training state.
Note that the training session created with a checkpoint state uses this state to store the entire
training state (including model parameters, its gradients, the optimizer states and the properties).
The TrainingSession does not hold a copy of the CheckpointState and as a result, it is required
that the checkpoint state outlives the lifetime of the training session.
Overrides SafeHandle.IsInvalid
returns true if handle is equal to Zero
Load a checkpoint state from a directory on disk into checkpoint_state.
This function will parse a checkpoint directory, pull relevant files and load the training
state into the checkpoint_state. This checkpoint state can then be used to create the
training session by instantiating the TrainingSession. By doing so, the training
session will begin or resume training from the given checkpoint state.
Absolute path to the checkpoint directory.
CheckpointState object which holds the state of the training session parameters.
Save the given state to a checkpoint directory on disk.
This function serializes the provided checkpoint state to a directory on disk.
This checkpoint can later be loaded by invoking CheckpointState.LoadCheckpoint to begin or resume
training from this snapshot of the state.
The checkpoint state to save.
Absolute path to the checkpoint directory.
Flag to indicate whether to save the optimizer state or not.
Adds or updates the given int property to/in the checkpoint state.
Runtime properties such as epoch, training step, best score, and others can be added to the checkpoint
state by the user by calling this function with the corresponding property name and value.
The given property name must be unique to be able to successfully add the property.
Name of the property being added or updated.
Property value associated with the given name.
Adds or updates the given float property to/in the checkpoint state.
Runtime properties such as epoch, training step, best score, and others can be added to the checkpoint
state by the user by calling this function with the corresponding property name and value.
The given property name must be unique to be able to successfully add the property.
Name of the property being added or updated.
Property value associated with the given name.
Adds or updates the given string property to/in the checkpoint state.
Runtime properties such as epoch, training step, best score, and others can be added to the checkpoint
state by the user by calling this function with the corresponding property name and value.
The given property name must be unique to be able to successfully add the property.
Name of the property being added or updated.
Property value associated with the given name.
Gets the property value associated with the given name from the checkpoint state.
Gets the property value from an existing entry in the checkpoint state. The property must
exist in the checkpoint state to be able to retrieve it successfully.
Name of the property being retrieved.
Property value associated with the given property name.
Updates the data associated with the model parameter in the checkpoint state for the given parameter name.
This function updates a model parameter in the checkpoint state with the given parameter data.
The training session must be already created with the checkpoint state that contains the parameter
being updated. The given parameter is copied over to the registered device for the training session.
The parameter must exist in the checkpoint state to be able to update it successfully.
Name of the parameter being updated.
The parameter data that should replace the existing parameter data.
Gets the data associated with the model parameter from the checkpoint state for the given parameter name.
This function retrieves the model parameter data from the checkpoint state for the given parameter name.
The parameter is copied over to the provided OrtValue. The training session must be already created
with the checkpoint state that contains the parameter being retrieved.
The parameter must exist in the checkpoint state to be able to retrieve it successfully.
Name of the parameter being updated.
The parameter data that is retrieved from the checkpoint state.
Overrides SafeHandle.ReleaseHandle() to properly dispose of
the native instance of CheckpointState
always returns true
Creates an instance of OrtSession with provided parameters
checkpoint string path
(Output) Loaded OrtCheckpointState instance
Creates an instance of OrtSession with provided parameters
OrtCheckpointState instance to save
Checkpoint string path
Flag indicating whether to save the optimizer state.
Creates an instance of OrtSession with provided parameters
Native OrtEnv instance
Native SessionOptions instance
Loaded OrtCheckpointState instance
model string path
model string path
model string path
(Output) Created native OrtTrainingSession instance
This class defines utility methods for training.
Use this function to generate reproducible results. It should be noted that completely
reproducible results are not guaranteed.
Manual seed to use for random number generation.
Trainer class that provides training, evaluation and optimizer methods for training an ONNX model.
The training session requires four training artifacts
- The training onnx model
- The evaluation onnx model (optional)
- The optimizer onnx model
- The checkpoint directory
These artifacts can be generated using the `onnxruntime-training` python [utility](https://github.com/microsoft/onnxruntime/blob/main/orttraining/orttraining/python/training/onnxblock/README.md).
This is an IDisposable class and it must be disposed of
using either an explicit call to Dispose() method or
a pattern of using() block. If this is a member of another
class that class must also become IDisposable and it must
dispose of TrainingSession in its Dispose() method.
A pointer to an underlying native instance of OrtTrainingSession
Create a training session that can be used to begin or resume training.
This constructor instantiates the training session based on the env and session options provided that can
begin or resume training from a given checkpoint state for the given onnx models.
The checkpoint state represents the parameters of the training session which will be moved
to the device specified by the user through the session options (if necessary).
Training states that the training session uses as a starting point for training.
Model to be used to perform training.
Model to be used to perform evaluation.
Model to be used to perform weight update.
Create a training session that can be used to begin or resume training.
This constructor instantiates the training session based on the env and session options provided that can
begin or resume training from a given checkpoint state for the given onnx models.
The checkpoint state represents the parameters of the training session which will be moved
to the device specified by the user through the session options (if necessary).
Training states that the training session uses as a starting point for training.
Model to be used to perform training.
Model to be used to perform weight update.
Create a training session that can be used to begin or resume training.
This constructor instantiates the training session based on the env and session options provided that can
begin or resume training from a given checkpoint state for the given onnx models.
The checkpoint state represents the parameters of the training session which will be moved
to the device specified by the user through the session options (if necessary).
SessionOptions that the user can customize for this training session.
Training states that the training session uses as a starting point for training.
Model to be used to perform training.
Model to be used to perform evaluation.
Model to be used to perform weight update.
Computes the outputs of the training model and the gradients of the trainable parameters for the given inputs
This function performs a training step that computes the outputs of the training model and the gradients
of the trainable parameters for the given inputs. The train step is performed based on the training model
that was provided to the training session.
The TrainStep method is equivalent of running forward propagation and backward propagation in a single
step.
The gradients computed are stored inside the training session state so they can be later consumed
by the OptimizerStep function.
The gradients can be lazily reset by invoking the LazyResetGrad function.
Specify a collection of that indicates the input values to the training model.
Specify a collection of that indicates the output values of the training model.
Computes the outputs of the training model and the gradients of the trainable parameters for the given inputs
This function performs a training step that computes the outputs of the training model and the gradients
of the trainable parameters for the given inputs. The train step is performed based on the training model
that was provided to the training session.
The TrainStep method is equivalent of running forward propagation and backward propagation in a single
step.
The gradients computed are stored inside the training session state so they can be later consumed
by the OptimizerStep function.
The gradients can be lazily reset by invoking the LazyResetGrad function.
Specify for step.
Specify a collection of that indicates the input values to the training model.
Specify a collection of that indicates the output values of the training model.
Computes the outputs of the training model and the gradients of the trainable parameters for the given inputs
This function performs a training step that computes the outputs of the training model and the gradients
of the trainable parameters for the given inputs. The train step is performed based on the training model
that was provided to the training session.
The TrainStep method is equivalent of running forward propagation and backward propagation in a single
step.
The gradients computed are stored inside the training session state so they can be later consumed
by the OptimizerStep function.
The gradients can be lazily reset by invoking the LazyResetGrad function.
Specify a collection of that indicates the input values to the training model.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Computes the outputs of the training model and the gradients of the trainable parameters for the given inputs
This function performs a training step that computes the outputs of the training model and the gradients
of the trainable parameters for the given inputs. The train step is performed based on the training model
that was provided to the training session.
The TrainStep method is equivalent of running forward propagation and backward propagation in a single
step.
The gradients computed are stored inside the training session state so they can be later consumed
by the OptimizerStep function.
The gradients can be lazily reset by invoking the LazyResetGrad function.
Specify for step.
Specify a collection of that indicates the input values to the training model.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
This function performs a training step that computes the outputs of the training model and the gradients
of the trainable parameters for the given OrtValue inputs. The train step is performed based on the training model
that was provided to the training session.
The TrainStep method is equivalent of running forward propagation and backward propagation in a single
step.
The gradients computed are stored inside the training session state so they can be later consumed
by the OptimizerStep function.
The gradients can be lazily reset by invoking the LazyResetGrad function.
Example usage:
using OrtValue x = OrtValue.CreateTensorValueFromMemory(...);
using OrtValue label = OrtValue.CreateTensorValueFromMemory(...);
List{OrtValue} inputValues = new List{OrtValue} { x, label };
using (var loss = trainingSession.TrainStep(inputValues))
{
// process output values
}
Specify a collection of that indicates the input values to the training model.
Output Tensors in a Collection of NamedOnnxValue. User must dispose the output.
Convert native OrtValue handles to OrtValue instances
in an exceptions safe manner.
Reset the gradients of all trainable parameters to zero lazily.
This function sets the internal state of the training session such that the gradients of the trainable
parameters in the OrtCheckpointState will be scheduled to be reset just before the new gradients are
computed on the next invocation of the next TrainStep.
Computes the outputs for the eval model for the given inputs
This function performs an eval step that computes the outputs of the eval model for the given inputs.
The eval step is performed based on the eval model that was provided to the training session.
Specify a collection of that indicates the input values to the eval model.
Specify a collection of that indicates the output values of the eval model.
Computes the outputs for the eval model for the given inputs
This function performs an eval step that computes the outputs of the eval model for the given inputs.
The eval step is performed based on the eval model that was provided to the training session.
Specify for step.
Specify a collection of that indicates the input values to the eval model.
Specify a collection of that indicates the output values of the eval model.
This function performs an eval step that computes the outputs of the eval model for the given inputs.
Inputs are expected to be of type OrtValue. The eval step is performed based on the eval model that was
provided to the training session.
Example usage:
using OrtValue x = OrtValue.CreateTensorValueFromMemory(...);
using OrtValue label = OrtValue.CreateTensorValueFromMemory(...);
List{OrtValue} inputValues = new List{OrtValue} { x, label };
using (var loss = trainingSession.EvalSteps(inputValues))
{
// process output values
}
Specify a collection of that indicates the input values to the eval model.
Sets the learning rate for this training session.
This function allows users to set the learning rate for the training session. The current
learning rate is maintained by the training session and can be overwritten by invoking
this function with the desired learning rate. This function should not be used when a valid
learning rate scheduler is registered. It should be used either to set the learning rate
derived from a custom learning rate scheduler or to set a constant learning rate to be used
throughout the training session.
Please note that this function does not set the initial learning rate that may be needed
by the predefined learning rate schedulers. To set the initial learning rate for learning
rate schedulers, please look at the function RegisterLinearLRScheduler.
Desired learning rate to be set.
Gets the current learning rate for this training session.
This function allows users to get the learning rate for the training session. The current
learning rate is maintained by the training session, and users can query it for the purpose
of implementing their own learning rate schedulers.
float representing the current learning rate.
Registers a linear learning rate scheduler for the training session.
Register a linear learning rate scheduler that decays the learning rate by linearly updated
multiplicative factor from the initial learning rate set on the training session to 0. The decay
is performed after the initial warm up phase where the learning rate is linearly incremented
from 0 to the initial learning rate provided.
Number of warmup steps
Number of total steps
Initial learning rate
Update the learning rate based on the registered learning rate scheduler.
Takes a scheduler step that updates the learning rate that is being used by the training session.
This function should typically be called before invoking the optimizer step for each round,
or as determined necessary to update the learning rate being used by the training session.
Please note that a valid predefined learning rate scheduler must be first registered to invoke this function.
Performs the weight updates for the trainable parameters using the optimizer model.
This function performs the weight update step that updates the trainable parameters such that they
take a step in the direction of their gradients (gradient descent). The optimizer step is performed
based on the optimizer model that was provided to the training session.
The updated parameters are stored inside the training state so that they can be used by the next
TrainStep function call.
Performs the weight updates for the trainable parameters using the optimizer model.
This function performs the weight update step that updates the trainable parameters such that they
take a step in the direction of their gradients (gradient descent). The optimizer step is performed
based on the optimizer model that was provided to the training session.
The updated parameters are stored inside the training state so that they can be used by the next
TrainStep function call.
Specify for step.
Export a model that can be used for inferencing.
If the training session was provided with an eval model, the training session can generate
an inference model if it knows the inference graph outputs. The input inference graph outputs
are used to prune the eval model so that the inference model's outputs align with the provided outputs.
The exported model is saved at the path provided and can be used for inferencing with InferenceSession.
Note that the function re-loads the eval model from the path provided to TrainingSession
and expects that this path still be valid.
Path where the inference model should be serialized to.
Names of the outputs that are needed in the inference model.
Returns a contiguous buffer that holds a copy of all training state parameters
Whether to only copy trainable parameters or to copy all parameters.
Loads the training session model parameters from a contiguous buffer
Contiguous buffer to load the parameters from.
Whether to only load trainable parameters or to load all parameters.
Retrieves the names of the user outputs for the training and eval models.
Whether the training model output names are requested or eval model output names.
Retrieves the names of the user inputs for the training and eval models.
Whether the training model input names are requested or eval model input names.
Other classes access
Finalizer.
IDisposable implementation
IDisposable implementation
true if invoked from Dispose() method