DeanoC thoughts and musings

The need for speed Coming from a real-time world (games and graphics), current machine learning training is a shock. Even small simple training tasks take a  long  time, which gives me plenty of time to think about the performance of my graphs. Currently most of the open source deep learning stacks consist of a C++ and Cuda back-end driven by a Python/R/Lua front end. The Cuda code tends to be fairly generic, which makes sense from an HPC and experimental point of view.  Its important to note HPC stuff is very optimised but tends to really on standard interfaces with relatively custom generic back-ends. For example BLAS is a old FORTRAN standard for linear algebra, it  has numerous back-ends included optimised x64, Cuda, OpenCL etc.  However it only accelerates the tensor multiplies in a few data formats, other more data specific optimisation like overlapping conversions isn't in its remit. Its quite a different optimisation strategy from the rea...

Its a lie because it implies there is a 'premature' aspect to writing code, that you shouldn't worry about performance or power at some phase of the research/project. Its simple not true, there are time when its not the greatest priority but thinking about how it affects these areas is never wasted. At some fundamental level, software is something that takes power (electricity) into math. Anytime you don't do it optimally you waste power.  The only time that doesn't matter is when you have a small problem and lots of power, in practise few interesting problems are lucky to get away with that. If you work in machine learning or big data, the likely limit to what you can do is related to how many processors (wether they are CPU, GPU or FPGA) you can throw at the problem. Assuming you can't scale infinitely, then the results you can get out of a finite set of HW will largely be determined by how performant your code is. When you've designed your latest ...