nimxla/nn

Search:
Group by:
Source   Edit  

The nn module provides higher level functions for building neural network models using the nimxla graph API.

Types

AdamOptimizer = ref object of Optimizer
Source   Edit   
ChainedScheduler = ref object of Scheduler
Runs a sequence of schedulers in series Source   Edit   
CosineAnnealingLR = ref object of Scheduler
Source   Edit   
InitFunc = proc (c: Client; dims: openArray[int]; dtype: DataType; rng: var Rand): Buffer
InitFunc defines a function used to initialise a learnable parameter. Source   Edit   
LinearLR = ref object of Scheduler
Source   Edit   
Module = object
  variables*: OrderedTable[string, Variable]
  outputs*: seq[string]
  forward*: proc (x: Node; training: bool; output: var Outputs): Node
  info*: string
A module holds the state for a set of variables together with a forward computation which depends on an input value. The training flag is set to true when in training mode. If the module saves internal state then this is added to the output list at compile time. each forward call. Source   Edit   
Optimizer = ref object of RootRef
  state*: Params
Optimizer function to update the model weights. Subclasses should implement the step method Source   Edit   
Outputs = object
List of named output parameters Source   Edit   
Scheduler = ref object of RootRef
Scheduler updates the learning rate after each epoch Source   Edit   
SGDOptimizer = ref object of Optimizer
Source   Edit   
StepLR = ref object of Scheduler
Source   Edit   
Variable = object
  name*: string
  data*: Buffer
  calcGrad*: bool
A Variable is a learnable parameter associated with a module. Source   Edit  

Procs

proc `$`(m: Module): string {....raises: [], tags: [], forbids: [].}
Source   Edit   
proc `$`(s: Scheduler): string {....raises: [Exception], tags: [RootEffect],
                                 forbids: [].}
Source   Edit   
proc add(m: var Module; modules: varargs[Module]) {....raises: [], tags: [],
    forbids: [].}
Add variables from modules to m. Source   Edit   
proc compileTest(c: Client; m: Module; input: Node; softmax = false): Executable {.
    ...raises: [Exception, BuilderError, XLAError, ValueError], tags: [RootEffect],
    forbids: [].}
Build the execution graph for the given module and compile it to an executable. The executable returns the output from m.forward(input) If softmax is set then apply softmax function to the output. Source   Edit   
proc compileTrain(c: Client; m: Module; input: Node;
                  lossFn: proc (y: Node): Node): Executable {.
    ...raises: [Exception, ValueError, KeyError, BuilderError, XLAError],
    tags: [RootEffect], forbids: [].}
Build the execution graph for the given module and compile it to an executable. The executable returns a tuple with the following outputs:
  • pred: result of m.forward(input)
  • loss: result of lossFn(pred)
  • <v1.name>_grad, ...: gradients for each input variable with respect to the loss
Source   Edit   
proc constantInit(value: SomeFloat): InitFunc
Create a new buffer with a constant value. Source   Edit   
proc dropout(a: Node; ratio: float; training: bool; normalize = true): Node {.
    ...raises: [Exception, BuilderError], tags: [RootEffect], forbids: [].}
Dropout randomly replaces ratio fraction of input values with zeros when training is true else if a no-op in test mode. If normalize is set then the output is scaled by 1 / (1-ratio) Source   Edit   
proc format(val: float): string {....raises: [], tags: [], forbids: [].}
Source   Edit   
proc getParams(m: Module; params: var Params) {....raises: [], tags: [],
    forbids: [].}
Add model variables to the params table Source   Edit   
proc glorotInit(gain = 1.0): InitFunc {....raises: [], tags: [], forbids: [].}

Initialisation function with values set in uniform range from

 -gain*sqrt(6/(nin+nout)) to gain*sqrt(6/(nin+nout))
where nin are effective number of inputs and outputs. Also known as Xavier uniform init.

Source   Edit   
proc gradNames(m: Module): seq[string] {....raises: [], tags: [], forbids: [].}
List of all the variable gradient output names for this module. Source   Edit   
proc heInit(mean = 0.0; gain = 2.0; fanOut = false): InitFunc {....raises: [],
    tags: [], forbids: [].}

Initialisation function with values from normal distrubution with std deviation as

  sqrt(gain/nin) or sqrt(gain/nout) if fanOut is set
where nin are effective number of inputs and outputs. Gain of 2 is suitable for Relu non-linearity. Also known as Kaiming normal init.

Source   Edit   
proc init(s: Scheduler; c: Client; epoch: int) {.
    ...raises: [ValueError, Exception], tags: [RootEffect], forbids: [].}
Called when restoring state from checkpoint Source   Edit   
proc initBatchNorm(c: Client; rng: var Rand; id: string; numFeatures: int;
                   momentum: float = 0.1; epsilon: float = 0.00001;
                   weights = constantInit(1.0); biases = constantInit(0.0);
                   dtype = F32): Module {....raises: [ValueError, Exception],
    tags: [RootEffect], forbids: [].}
Create a new 2d batch normalization layer. The input should be in channels last format. numFeatures is the number of channels. The momentum parameter is used to compute the exponential running average of the mean and variance. epsilon is the value added to the denominator to avoid divide by zero. Learnable weight and bias parameters are initialised with the weights and biases initialization functions. Source   Edit   
proc initConv2d(c: Client; rng: var Rand; id: string;
                inChannels, outChannels: int; kernelSize: Opt2d;
                strides: Opt2d = 1; padding: Pad2d = pad(0);
                dilation: Opt2d = 1; groups = 1; weights = heInit();
                biases = constantInit(0.0); dtype = F32): Module
Create a new 2 dimensional convolution layer with parameters in channels last format. Weight parameters are initialised using the weights function. If biases is not nil then bias parameters are initialised using this function and added to the output. Source   Edit   
proc initConv2dBatchNorm(c: Client; rng: var Rand; id: string;
                         inChannels, outChannels: int; kernelSize: Opt2d;
                         strides: Opt2d = 1; padding: Pad2d = pad(0);
                         dilation: Opt2d = 1; groups = 1; momentum: float = 0.1;
                         epsilon: float = 0.00001; weights = heInit();
                         dtype = F32): Module
Create a 2 dimensional convolution layer with no bias followed by batch normalisation weights is the initialisation for the conv layer. The batchNorm layer will use default initialisation. Source   Edit   
proc initLinear(c: Client; rng: var Rand; id: string; nin, nout: int;
                weights = heInit(); biases = constantInit(0.0); dtype = F32): Module {.
    ...raises: [ValueError, Exception], tags: [RootEffect], forbids: [].}
Create a new fully connected linear layer with the given unique id and number of inputs and outputs. Weight parameters are initialised using the weights function. If biases is not nil then bias parameters are initialised using this function and added to the output. Source   Edit   
proc initRandom(seed: int64 = 0): Rand {....raises: [ValueError],
    tags: [TimeEffect], forbids: [].}
Initialize random generator. Picks a random seed from current time if seed == 0. Source   Edit   
proc learnableVars(m: Module): seq[Variable] {....raises: [], tags: [], forbids: [].}
List of all learnable variables for which we calc the gradients. Source   Edit   
proc learningRate(optim: Optimizer): float {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}
Source   Edit   
proc newChainedScheduler(schedulers: varargs[Scheduler]): ChainedScheduler {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}
Create a ChainedScheduler to run a number of schedulers one after the other. Source   Edit   
proc newCosineAnnealingLR(optim: var Optimizer; tMax: int; lrMin = 0.0): CosineAnnealingLR {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}

Create cosine annealing learning rate scheduler such that

lr = lrMin + (lrMax - lrMin)/2 * (1 + cos(pi * t/tMax))
where t is the current epoch and lrMax is the initial learning rate.

Source   Edit   
proc newLinearLR(optim: var Optimizer; epochs: int;
                 startFactor, endFactor: float): LinearLR {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}
Create LinearLR scheduler which linearly decays the learning rate from startFactor at the first epoch to endFactor after epochs. Source   Edit   
proc newStepLR(optim: var Optimizer; stepSize: int; gamma = 0.1): StepLR {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}
Create StepLR scheduler which multiplies the learning rate by gamma after each stepSize epochs. Source   Edit   
proc newVariable(c: Client; name: string; dims: openArray[int]; init: InitFunc;
                 rng: var Rand; calcGrad = true): Variable {.
    ...raises: [Exception], tags: [RootEffect], forbids: [].}
Allocate a new F32 variable with the given name and shape and initialise values. Source   Edit   
proc normalInit(mean = 0.0; stddev = 1.0): InitFunc {....raises: [], tags: [],
    forbids: [].}
Create a new buffer with random values chosen from a normal distrubution. Source   Edit   
proc optimAdam(c: Client; m: Module; learnRate: float; weightDecay = 0.0;
               beta1 = 0.9; beta2 = 0.999; eps = 1e-8): Optimizer {.
    ...raises: [Exception, BuilderError, XLAError, ValueError], tags: [RootEffect],
    forbids: [].}
Source   Edit   
proc optimAdamW(c: Client; m: Module; learnRate: float; weightDecay = 0.0;
                beta1 = 0.9; beta2 = 0.999; eps = 1e-8): Optimizer {.
    ...raises: [Exception, BuilderError, XLAError, ValueError], tags: [RootEffect],
    forbids: [].}
Source   Edit   
proc optimSGD(c: Client; m: Module; learnRate: float; weightDecay = 0.0;
              momentum = 0.0; nesterov = false): Optimizer {.
    ...raises: [Exception, BuilderError, XLAError, ValueError], tags: [RootEffect],
    forbids: [].}
Source   Edit   
proc param(b: Builder; p: Variable; suffix = ""): Node {.
    ...raises: [Exception, BuilderError], tags: [RootEffect], forbids: [].}
Create a new parameter node from this Variable Source   Edit   
proc setLearningRate(optim: var Optimizer; c: Client; lr: float) {.
    ...raises: [XLAError], tags: [], forbids: [].}
Update learning rate parameter Source   Edit   
proc setParams(m: var Module; params: Params) {....raises: [KeyError], tags: [],
    forbids: [].}
Update provided parameter list - e.g. from output from optimizer Source   Edit   
proc setVars(m: var Module; vars: varargs[Variable]) {....raises: [], tags: [],
    forbids: [].}
Assign variables to module Source   Edit   
proc step(optim: var Optimizer; params: Params): Params {.
    ...raises: [KeyError, XLAError, ValueError, Exception], tags: [RootEffect],
    forbids: [].}
Called after each batch to update the model parameters. Source   Edit   
proc step(s: Scheduler; c: Client) {....raises: [Exception], tags: [RootEffect],
                                     forbids: [].}
Called after each epoch to update the learning rate Source   Edit   
proc uniformInit(min = 0.0; max = 1.0): InitFunc {....raises: [], tags: [],
    forbids: [].}
Create a new buffer with uniform random values from min to max. Source   Edit   
proc update(m: var Module; params: Params) {....raises: [KeyError], tags: [],
    forbids: [].}
Update output parameter values Source   Edit   
proc varNames(m: Module): seq[string] {....raises: [], tags: [], forbids: [].}
List of all the learnable variable names defined for this module. Source   Edit  

Methods

method `$`(optim: AdamOptimizer): string {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}
Source   Edit   
method `$`(optim: Optimizer): string {.base, ...raises: [ValueError], tags: [],
                                       forbids: [].}
Source   Edit   
method `$`(optim: SGDOptimizer): string {.
    ...raises: [ValueError, XLAError, KeyError], tags: [], forbids: [].}
Source   Edit  

Made with Nim. Generated: 2023-08-09 12:14:55 UTC