Skip to content

NOTES.md

Latest commit

 

History

History
274 lines (249 loc) · 9.95 KB

NOTES.md

File metadata and controls

274 lines (249 loc) · 9.95 KB
Raw

Notes

Tensor

Todo

Done

  • use vectors to store data
  • separate the view and data parts of tensor to not have to copy everything?
    • use pointers?
    • cant use (mut) reference because you need to initialize data somehow, it must belong to the tensor that created it
    • tensor must be able to mutate data to do in-place operations
    • option to do in-place or copy
    • so don't need to separate data and tensor, but keep in mind for future if you want to do more (or multiple) inplace ops
    • make two, slice and slice_mut
    • just make copies immutable to make it simple
  • right now, calc_shape_from_slice will slice a tensor of shape [2, 2, 2] to [1, 1, 1] if slicing it on [[0, 1], [0, 1], [0, 1]]
    • this means that strides for 3+ d tensors can be wrong
  • vector's overhead shouldn't be much
  • maybe set vec to not allocate extra memory
  • vec is contigous in memory
  • naive slicing is too slow
  • numpy sometimes makes a copy when slicing to keep stuff contigous
  • naive approach slows down with size
  • it looks like numpy will make a copy when slicing
  • dont' use [start:stop] syntax for rust, use .slice(!vec[start:stop])
  • you should be able to do: will make copies to make it simple
  • numpy notes
  • ndarray heap (https://users.rust-lang.org/t/ndarray-stack-and-heap-memory-and-overhead/25254)
  • pre-allocate size of new vectors
  • add checks for slicing and tensor creation
  • validation should happen at initialization
  • fix error handling and structure
  • panic!("Invalid slice")
  • should panicing be done in main?
  • use smarter indexing
    • -1 for all values
    • not needing to say either first or last vals
    • slicing with an empty slice will return a copy of the orignal tensor
  • reduce number of &[..]
  • reshape
  • benchmarks for previous stuff
  • broadcasting
  • ops
    • should be like let c = &a + &b; and let c = l2::add(a, b);
    • elementwise ops are comparable to numpy until ~ 4096x4096
    • other should be reference
    • ops won't return Result<Tensor, TensorError>
      • that would mean that you would need to run let x = (a + b).unwrap();
      • will panic instead
    • all ops are tensor-tensor ops
    • element-wise ops
  • self-ops
    • pow
    • sqrt
    • exp
    • log
      • e
      • 10
    • abs
    • sin
    • cos
    • tan
  • other ops
    • dim ops are slower than numpy (30us vs 300us), numpy seems to cache stuff
    • argmax and argmin return f32 tensors
    • over dim or all dims
    • sum
    • mean
    • max
    • min
    • argmax
    • argmin
  • use enum for ops?
  • matmul
    • about 100x slower than numpy
    • wont implement broadcasting on matmul
    • batch matmul
  • check errors
  • concat
  • transpose
  • clone
  • normal
  • uniform
  • autodiff
    • accumulate gradients
    • hold reference to parents
      • lhs and rhs
    • hold gradient
    • know grad functions for each op
    • account for creators
    • do tensor-tensor op without grad tracking for backend
    • derivative is RefCell<Option<Tensor>>
    • borrow_mut() on derivative and assign to it
    • use Rc::new(Tensor::new(Rc::clone(&t))), so references can be to more than one tensor?
    • dont need to use option?
      • no
    • use a wrapper for grad tracking tensors?
      • save memory on normal tensors
    • mark nodes as evaluated?
      • prevent having to recurse through shared graph multiple times
      • topological sort
        • done
      • in backward mutate lhs and rhs parent's grad not own
        • works
   a
  / \
 *   *
/     \
b      c
\     /
 +   +
  \ /
   d

da =  dd/db + dd/dc
but this will recompute the backwards pass for all the graph above a
topological sorting accumulates gradients for a before going further up the computation graph
  • printing of tensors and graph
  • combine 1,2,3,4d ops into one function
  • figure out macros and crates
  • ops
    • div
    • pow
    • sqrt
    • exp
    • log10
    • log
    • abs
    • sin
    • cos
    • tan
    • slice
      • allocates a small (1 element) tensor to satisfy match arms
    • transpose
      • allocates a small (1 element) tensor to satisfy match arms
    • view
    • concat
    • sum
    • mean
  • figure out if you want to use tensor ops in backward or vec ops
    • decide what to do with new_with_parents
    • used tensor ops
  • clear gradient
  • blas
  • autograd
    • benchmarks for each operator
      • use correct shapes
      • compare to pytorch and ndarray
    • only 100s of ns used for topo sort, rest is calling backwards
  • cargo.rs

wont do

  • change col-major to row-major
  • impl iterator
  • replace indices slice with enum
  • tensor::new shouldn't return result
  • use enum to store diff between two types of indices
    • [start, end)
    • -1
  • prevent having to reallocate memory on each backwards pass
    • clear unneeded memory as soon as you can?
  • impl == on tensors
  • derivative vec for broadcasted mul to a one-element tensor is 6 elements long
    • problem with derivatives when broadcasting
  • ops
    • matmul
      • 3 and 4d backwards don't work yet because transpose should only work on two dims
    • max
    • min
    • argmax
    • argmin
  • fix transpose
  • redo error handling
  • const generics for compile time errors
  • fix blas ci

Notes

  • L2 is competitive with numpy for small tensors
  • L2 copies slices since thats whats needed for autograd
    • by default, numpy returns a view
  • takes about 6s to slice all elements from a 64x64x64x64 tensor
  • speed of slicing/allocating cannot be optimized. Numpy takes about 2x the time that l2 does because l2 will always copy a slice. Numpy's native slices are views, but copies are needed for autograd.

Resources

backprop

Autodiff

Rust

SIMD