In January, we introduced you to Torben’s Corner – a resource wiki created and maintained by Jacket programming guru, Torben Larsen at Aalborg University in Denmark.  Many Jacket programmers have gained valuable insights from Torben’s Corner, including GPU performance charts, coding guidelines, special tricks.

Since January, many wonderful additions have been added to Torben’s Corner.  We think you will find value in not only this new information but the entire resource.  Here is a quick summary of the most recent additions with links to the information:

Benchmarking Update

Torben’s Corner maintains a long list of benchmarks specifically detailing speedups of Jacket relative to standard MATLAB. This became an enormous task due to the sheer number of functions supported by Jacket and MATLAB.

Therefore, a new benchmarking library has been designed. This library currently consists of 30 benchmarked functions and more will be added over time. These benchmarks measure the speedups of Jacket versus CPU-based MATLAB for matrix/vector and single/double precision. All results are gathered in three automatically generated outputs:

1. a .mat file with all the measured data,
2. a .wki file which is a Wiki table ready to be inserted in a Wiki page, and
3. a Latex table with everything needed to produce a beautiful Latex table just by the press of a button.

The library is freely available for anyone to use – with the standard requirements for open access software. You can read more and download the benchmark. The full set of benchmarks takes 1.5-2 hours to complete.

Installing on Ubuntu

A guide on how to install Jacket on Ubuntu, including installation instructions for Ubuntu, GPU Drivers, MATLAB, Jacket, and Dropbox.

Running on a plain MATLAB install

A procedure and library of functions enabling Jacket programmers to write GPU code which will run with a plain CPU-only MATLAB installation. This is very important when programmers want to distribute their toolboxes, etc. Of course, without the GPU, the code will run slower, but with this procedure, your code will run without a GPU.

CPU settings that affect performance

A small post on how the CPU performance settings affect the floating point performance of the CPU and Jacket.

Jacket performance study

A detailed study on the floating point performance of Jacket. This is ongoing work and new results are continually being added.

———-

Thanks to Torben and his team for these contributions and hope all Jacket programmers can leverage this most recent work and the resource as a whole.

GTC is coming up…

The GPU Technology Conference (GTC) starts later this month and is sure to generate a new level of excitement and energy around GPU computing.  The conference includes over 250 technology sessions presented by industry, government, and academic technology leaders.  AccelerEyes is pleased to be well represented at this year’s conference by our technical leadership and a number of our customers.  If you plan to attend the conference be sure to include the sessions outlined below on your agenda.

In addition to being well represented, we are also flattered to see that others in the market have recognized that GPU Computing with MATLAB delivers clear productivity gains and that the performance improvements made possible by GPUs is a reality today.  Most notably, The MathWorks will share its vision and capabilities for GPU Computing with MATLAB during the conference, which should increase the visibility and demand for the technology worldwide.  We encourage everyone to attend the session to learn about their new offering.

AccelerEyes will be demoing Jacket at Table #56 and hope that you will stop by to see the latest and greatest Jacket technology during the conference.

Jacketized GTC Sessions

2132 – Accelerating Biologically Inspired Computer Vision Models

Join us for a discussion on applying commodity-server-based clusters and GPU-based clusters to simulating computer vision algorithms at a scale that approaches that of biological vision. We consider the limitations of each technology, survey approaches taken thus far, and suggest new hybrid models and programming frameworks to overcome current limitations and substantially improve performance.

2268 – Think Data-Parallel! Building Data-Parallel Code with M

Discover and leverage parallelism inherent in pre-existing codes. Often times, parallelism is hidden in seemingly serial programs. This is due obfuscation via indexing or looping wherein the parallelism is seemingly non-existent. Several real-world examples of seemingly serial code demonstrate simple, yet surprisingly effective rules for detecting potential parallelism.

For each example, learn how to express the code at a higher, more concise level in M by vectorizing computations. We give several canned techniques of vectorization for many common, and sometimes very difficult, use cases.

Learn how such vectorization concisely brings the parallelism of code to the forefront and transforms programs that might have been originally difficult to run on a SIMT device very suitable for execution on the GPU. GPU speedups will be shown utilizing Jacket.

2300 – High-Performance Compressive Sensing using Jacket

This talk will present the ongoing work that I am doing in the L1-optimization group at Rice University. The purpose of the work is to merge both compressive sensing, for image/signal reconstructions and GPU computation, using NVIDIA’s GPUs to enhance the technology of CS.

This talk will cover basic concepts in compressive sensing and the easy adaptation of operating on the GPU, in particular working with Jacket (by AccelerEyes). We will then cover some of our numerical experiments that encompass the use of different flavors of algorithms.

2201 – A Case Study of Accelerating Matlab Based Applications using GPUs

Learn how to accelerate Matlab based applications using GPUs. We cover a popular neuro-imaging software called SPM and show how to use CUDA and Jacket to speedup computationally intensive Matlab applications.

2271 – Compose CUDA Masterpieces! Write better, Leverage More

Not all CUDA code is created equally. Learn how to step up your CUDA game. Also, learn how to build large, multi-person CUDA projects for your organization.

In very clear descriptions, learn the difference between naïve GPU code, intermediate GPU code, and advanced GPU mastery. We show how careful construction of CUDA kernels can affect application performance.

We also discuss how Jacket tools greatly facilitate the development of CUDA-based projects.

Finally, we will debut the Jacket runtime’s new C/C++ library. With this library, the technical computing functions in Jacket’s MATLAB engine are made available in C/C++.

2100 – Hybrid GPU/Multicore Solutions for Large Linear Algebra Problems

Large linear algebra problems may be solved using recursive block decomposition in which GPUs efficiently compute the sub-blocks and multicore CPUs put the sub-blocks back together within a large shared memory space. This talk will present benchmark results for such a hybrid approach, implemented in Matlab® and using Jacket® to access the GPU compute power.

Following our recent Jacket v1.4 Fermi architecture release, many of you requested data comparing the new NVIDIA Fermi-based Tesla C2050 versus the older Tesla C1060.

Over the years, AccelerEyes has developed an extensive suite of benchmark MATLAB applications, which are included in every Jacket installation. Using this suite of tests, we compared performance of the C2050 vs C1060 and are pleased to report the results here. We hope this information will be useful to Jacket programmers.

All tests were run on the same standard workstation with Jacket 1.4. The only thing that changed was the actual GPU board. In every case the C2050 beat the C1060. Double-precision examples on the Fermi-based board outperformed the older board by 50% in every case and better than 2x in many cases.

Note: ECC was enabled on the Fermi boards

In addition to the standard Jacket examples, matrix multiplication with SGeMM and DGeMM was performed and plotted in the following charts. This matrix multiply implementation was developed in-house at AccelerEyes and outperforms both CUBLAS and Magma considerably, see MTIMES benchmarks. Special thanks to Torben Larsen for benchmarking results.

As we generate or receive more comparison data we will communicate results.

We are pleased to announce Jacket 1.4, with support for the latest NVIDIA graphics processing units based on the Fermi architecture (Tesla 20-series and GeForce GTX 4xx-series). NVIDIA’s release of the Fermi architecture brings with it 448 computational cores, increased IEEE-754 floating-point arithmetic precision, error-correcting memory for reliable computation, and enhanced memory caching mechanisms.

Highlights for Jacket 1.4 are as follows:

• Added support for the NVIDIA Fermi architecture (GTX400 and Tesla C2000 series)
  • Jacket DLA support for Fermi
• Dramatically improved the performance of Jacket’s JIT (Just-In-Time) compilation technology
  • Operations involving random scalar constants do not incur a recompile
  • Removed dependencies on MINGW and NVCC
• Logical indexing now supported for SUBSREF and SUBSASGN, e.g. B = A(A > x)
• MTIMES supports mixed types, no longer uses CUBLAS, and achieves better performance than CUBLAS
• SUM, MIN, MAX, ANY, ALL now supported over any number of columns, rows, or dimensions
• MIN, MAX indexed output now supported for complex single and complex double inputs
• SUM, MIN, MAX over columns is greatly accelerated; vectors accelerated too
• FIND performance improvements
• CONVN, BLKDIAG, DOT performance improvements
• CUMSUM now supported for matrices also
• SORT, CONVN now supported in double-precision
• HESS(A) and [P,H] = HESS(A) now supported (see Jacket DLA)
• LEGENDRE now supported
• Expanded GFOR support for:
  • MLDIVIDE, INV, HESS, MTIMES
  • FFT, FFT2, FFTN and inverses IFFT, IFFT2, IFFTN
• PCG now supported, this is a system solver that uses the Preconditioned Conjugate Gradient Method for dense matrices
• Image Processing Library now available. Direct access to the NVIDIA Performance Primitives (NPP) enabling new image processing functionality such as ERODE and DILATE.

The release notes are as follows:

See http://wiki.accelereyes.com/wiki/index.php/Release_Notes for full release notes.

AccelerEyes is focused on not only providing the most easy to use GPU programming platform for CUDA capable GPUs by leveraging the MATLAB® language, our engineering organization is always looking for ways to improve the performance of all areas in the Jacket platform. A case in point is some recent work with matrix multiplication, specifically (Single General Matrix Multiply) SGEMM, or MTIMES.  The Jacket 1.3 release was based on CUBLAS for matrix multiplication and given the importance of matrix multiplication to so many of our customers, we decided to find out if we could improve performance of the function.

Update: The new MTIMES routine in Jacket 1.4 has improved sigificantly since these benchmarks of the Release Candidate were taken. Have a look at the MTIMES Benchmarks wiki for up-to-date performance results, including Fermi (Tesla C2050) benchmarks!

The following chart illustrates the improvements we were able to make with our custom GEMM implementation in the 1.4 Release Candidate. The performance timings for SGEMM at various sizes of square matrices were performed, comparing the Jacket 1.3 Release and the Jacket 1.4 Release Candidate.

The graph above represents C=A*B timings for every NxN matrix from 10×10 to 4000×4000. Each data point is an average of 2 calls of MTIMES for each matrix size. Notice that Jacket 1.3 follows exactly 2 curves; the lower curve for certain multiples of 16 and 32, and the upper curve for everything else. The lower blue curve is attained by not using textures, as the data aligns and coalesces nicely in memory, and using other optimizations for certain data sizes. By switching to using only texture memory, Jacket 1.4 sacrifices some performance slightly in certain cases, but along with other optimizations, it brings faster consistent run-times, gives enhanced GFOR performance and enables native mixed matrix types support. In addition to texture memory, careful shared memory and register usage brings significant performance enhancements as well.

The above graph overlays a MATLAB R2009b 32-bit (CPU) benchmark of MTIMES.

The graph above represents C=A*B GFLOPS for every NxN matrix from 10×10 to 4000×4000.

The GFLOPS formula used:

Miscellaneous information regarding the Jacket installation and hardware used for testing:

>> ginfo
AccelerEyes Jacket v1.4.0rc2 (build 4971)
CUDA driver: 195.36.15, CUDA toolkit 3.0
Memory: 0 CPU-used, 23 GPU-used, 4034 GPU-free (in MB)
License Type: Designated Computer
License Features: jacket sdk mgl4 dla
Multi-GPU: Licensed for 4 GPUs
Detected CUDA-capable GPUs:
GPU0 Tesla C1060, 1265 MHz, 4095 MB VRAM, Compute 1.3 (single,double) (in use)
GPU1 GeForce 8400 GS, 896 MHz, 511 MB VRAM, Compute 1.1 (single)

$ cat /proc/cpuinfo
vendor_id      : GenuineIntel
model name    : Pentium(R) Dual-Core  CPU  E5200  @ 2.50GHz
cpu MHz        : 2499.934
cache size     : 2048 KB
cpu cores      : 2
…

Update
Below is an example of the SGEMM performance on a Tesla C2050 (Fermi) card
-  Jacket 1.3 vs. Jacket 1.4 Final Release.

References
* Volkov, V., and Demmel, J. W., Benchmarking GPUs to tune dense linear algebra, SC08.
* SGEMM code from Vasily Volkov.
* SGEMM code from Lung-Sheng Chien.
* See the MTIMES benchmarks wiki page for more info!

The research and engineering teams at AccelerEyes have prepared some exciting new additions for Jacket. These additions will enable you to get even more leverage out of NVIDIA GPUs for computing in MATLAB.  Over the past few years we’ve had the pleasure of working along side scientists and engineers using Jacket, and have learned a great deal about how people want to use MATLAB and GPUs. These new additions make significant progress towards enabling programmers to profile GPU computations and exert greater control over compilation and kernel execution inside Jacket.

In this post, we introduce GPROFILE and GCOMPILE for Jacket in anticipation of their upcoming release in the near future.  A brief overview of these two new functions follows.

GPROFILE: One-of-a-kind Profiling and Tuning Tool for GPUs and MATLAB
Historically, Jacket’s optimizations have occurred behind-the-scenes. Feedback from programmers on the Jacket Forums led us to dig in and develop a tool to enable users to monitor their scripts and see reports of runtime details that are unique to GPROFILE:

• timings between CPU and GPU for each GPU function,
• differences between the results of CPU and GPU functions (for easy numerics and debugging),
• unique timings per input size of CPU/GPU functions.

This diagnostic output will be available in the immediate future as:

• HTML reports viewable in a browser,
• Color coded reports on the MATLAB command line.

These reports immediately identify the areas of code that are taking advantage of the GPU, how much benefit the GPU is giving, and also identify areas of code that are slow and need attention. Coupled with the Jacket Tips and Code Vectorization resources, GPROFILE reports add a new level of high-performance tuning for Jacket codes.

GPROFILE Console Report

Below is an example session of work with GPROFILE.  To profile Jacket code, the profiler is first enabled with the GPROFILE ON command (the GUI version, gprofview, will be presented in a later blog post).  Then, the code to be profiled is executed – here we are running a simple k-means clustering script. GPROFILE REPORT (the base reporting mechanism on the console) gives a simple overview of performance on a per-command basis,

GPROFILE Console Report on a high end CPU (i950) versus a mid range GPU (280M)

Here, we see that, on average, all the calls to the functions SUBSREF, BSXFUN, TIMES, PERMUTE, EQ, and FIND on the GPU outperformed the CPU versions by varying factors as indicated by their green coloring.  Red lines indicate commands that, on average, performed worse on the GPU than the CPU; these are most likely caused by either minuscule timings (such as the case with SUBSASGN at the end of the list), odd memory arrangements (as in the case of MIN), or small data sizes (as in SUBSASGN on line 17 – the data size report is not shown for SUBSASGN).  In this case, GPROFILE is particularly useful as it identifies MIN as something that could be done differently to get the maximum work out of the GPU and in the future, GPROFILE will make suggestions on how to get the performance in such situations up.

Drilling down deeper by adding keywords (commands names, file names, and line numbers) to GPROFILE REPORT, information on a per-line basis is available and also on a per use case basis as shown on the last line of the screenshot.  Here, we see that TIMES was executed on two matrices of size 23 million by 5.  This information is very important for code development because data-parallel programs’ speed and algorithms depend on and are very sensitive to data size, arrangement, and shape.  Being able to drill down to CPU vs GPU timings on a per-data-size and per-use-case basis is another unique feature of GPROFILE that allows programmers to optimize algorithms down to each of their use-cases or otherwise better-leverage their GPUs by maximizing the funnels of data making their way through code.

The final unique feature of GPROFILE is the identification of lazily evaluated functions or functions that are compiled directly to PTX by Jacket at run-time as shown in the last column of the GPROFILE REPORT output.  By maximizing the use of these functions and chaining these together as much as possible, memory latency is minimized in the GPU code that Jacket runs.

GCOMPILE

GCOMPILE is a new feature that allows MATLAB developers to pre-compile sections of performance critical code. After isolating a critical section, developers format the code and pass it through GCOMPILE for analysis. The compiled function can then be applied with less overhead.

Let’s jump into an example. At present, one of the key missing features of GFOR are if-statements. With GCOMPILE, Jacket is able to handle such statements. For example, suppose you wanted to threshold an image and needed to use an if-statement. You need to use something like verbatim.m to push your critical function into a string for the compiler.

code = verbatim;
%{
  function out = main(in, threshold)
    if in > threshold
      out = threshold / 5;
    else
      out = sin(in);
    end
  end
%}
% compile and get function handle for later use
fn = gcompile(code);
thresh = 42;
for i = 1:n
  img = volume(:,:,i);
  out(:,:,i) = fn(img, thresh);  % apply the function to each image slice
end

The string code sample is essentially applied element-wise. In the main() function, the variable in contains a scalar element pulled from img while threshold is the same MATLAB scalar 42;

Another example kernel might look like this:

code = verbatim;
%{
function D = code(A, B, C)
D = A * C + cos(B);  % all element-wise
end
%}

Jacket v1.4 release candidates are rolling out with new on-the-fly compilation technology, GCOMPILE is the first of many extensions building on that technology. Please contact support if you would like to be included in the pre-release testing. GCOMPILE is scheduled for inclusion in the first release candidate after v1.4. The initial version only supports a subset of the M language.

Here are some other planned features, but be sure to tell us what you want in the comments section or on our forums.

• Read the functions from disk.
• Nested functions
• Reductions (e.g. sum/min/max/..)
• For-loops, subscripting, etc.

~Puyan Lotfi, ~Brett Lucey (gcompile)
~Gallagher Pryor, ~Joe Uhl (gprofile)

With IBM’s decision this week to integrate Tesla technology into it’s high performance computing line, there should be no doubt that GP-GPU computing is more than a fad, organizations solving technical problems are able to do them more productively and efficiently than ever before with GPUs.  AccelerEyes’ customers are experiencing this first hand with the Jacket product family as they are able to quickly and easily implement new or existing algorithms for GPUs and accomplish their technical needs much faster with substantial speed improvements.

Case in point, this week, AccelerEyes has released two case studies from customers that have used Jacket to transform their applications to GPU Computing with compelling results.

System Planning Corporation has implemented two different radar processing applications for the GPU with Jacket that have provided the insight they needed to fully understand how GPUs can benefit their customer requirements.  Learn more about these applications:

http://www.accelereyes.com/resources/radarprocessing

http://www.accelereyes.com/resources/radarformation

The Laboratory for Spectral Diagnosis at Northeastern University, under the leadership of Professor Max Diem, has a research focus on spectral diagnosis of disease and have leverage Jacket and GPUs to accelerate their research while also accelerating the time it takes to identify cancer and other illnesses.  Learn more at http://www.accelereyes.com/resources/spectroscopy

Jacket and NVIDIA GPUs continue to address and accelerate a growing list of science, engineering and analytical applications.  Additional case studies of application using Jacket can be found at http://www.accelereyes.com/successstories

If you’re a MATLAB user and want to apply your applications to NVIDIA GPUs for performance improvement but don’t want to write C, C++, or CUDA code, attend this seminar to learn more about Jacket for MATLAB – http://www.accelereyes.com/resources/junewebinar

In December of 2008, we did a blog post answering questions from customers and prospects about the use of OpenCL for Jacket.  If you have not reviewed that blog post to gain some insight into our progress you can access it here – http://blog.accelereyes.com/blog/2008/12/30/opencl/.

Some things have changed since that original post.  For example, NVIDIA now provides an OpenCL driver, toolkit, programming guide, and SDK examples.  Given the new tools available and the new Fermi hardware, we ran some tests on the Tesla c2050 to compare OpenCL performance to CUDA performance.  The Tesla C2050 is an amazing beast of a card, providing upto 512 Gigaflops of double precision arithmetic (at peak).

Before we present the benchmarks, we should comment on the programmability of OpenCL versus CUDA.  OpenCL is notably more difficult to program and debug than CUDA since OpenCL documentation, tools, and scientific computation libraries are still very limited.  Considering these handicaps, only a few matrix / vector operations were considered for this benchmark.  All the vector operations are modified versions of the SDK examples provided by NVIDIA.  All the tests were for single precision numbers.

Here are the results:

The results indicate that there is an overhead when using OpenCL with smaller data sizes, which seems to disappear at larger data sizes. Currently it is unknown whether the overhead is due to the time taken to launch a kernel in OpenCL or something else within the API.

This is our report of the current status of OpenCL relative to CUDA on the new NVIDIA hardware.  We continue to watch the progress of OpenCL, ATI, and other GPU computing initiatives.  Our focus is to deliver the best GPU computing platform on the planet for engineers, scientists, analysts, and students, and we guarantee that Jacket customers will always have the very best in GPU hardware choices for your applications.  As the GPU landscape continues to evolve, your Jacket code will simply get faster without you having to do anything new.  So, we invite you to sit back, relax, and enjoy watching your Jacket code scale with each new release!

Useful links:

1)  http://www.anandtech.com/show/2977/nvidia-s-geforce-gtx-480-and-gtx-470-6-months-late-was-it-worth-the-wait-/6

2)  http://www.pcgameshardware.com/aid,743498/Geforce-GTX-480-and-GTX-470-reviewed-Fermi-performance-benchmarks/Reviews/

3) http://www.appleinsider.com/articles/08/12/10/nvidia_pioneering_opencl_support_on_top_of_cuda.html

4)  http://www.sisoftware.co.uk/?d=qa&f=gpu_opencl&a=AMD

5)  http://unigine.blogspot.com/2010/02/cuda-vs-opencl-vs-directcompute.html

Over the last couple of months we have participated in some pretty amazing stories about dramatic speed ups for customer M-codes. For example, one customer went from 400 minutes of run time to 20 seconds and then further optimization dropped their runtime from 20 seconds to 65 milliseconds! This signifies a greater than 1000x performance improvement from the original code. Certainly, a runtime difference like that can make the difference in solving a problem versus not even attempting the effort.

When performance is critical to your problem, software platforms like Jacket can help to leverage hardware like GPUs, but additionally, vectorization of MATLAB code can and will make a huge difference in the runtimes of your scripts whether operating in single threaded, multi-threaded, CPU cluster or GPU computing mode. Vectorization may take time and effort but when performance is really important it is more than likely worth the effort.

MATLAB, parallel computing, and GPU computing all perform best on vectorized code. They all take advantage of the inherent parallelism of the M-language which is extremely powerful when utilized wisely. There are numerous sources of information available on the internet to learn about vectorization and to obtain vectorization examples,

Mathworks – Code Vectorization Guide
http://www.mathworks.com/support/tech-notes/1100/1109.shtml
MATLAB Tutorial at Cyclismo
http://www.cyclismo.org/tutorial/matlab/vector.html
Improving the Speed of MATLAB Calculations – Portland State University
http://web.cecs.pdx.edu/~gerry/MATLAB/programming/performance.html
matlab tips and tricks and …
http://www.ee.columbia.edu/~marios/matlab/matlab_tricks.html

In addition to these generally available resources to help in vectorizing M-code, AccelerEyes has begun (just begun) to build a library of vectorization examples that we are confident will help Jacket users dramatically improve their code. We will continue to build on this library of examples so bookmark the wiki page and visit often.

http://wiki.accelereyes.com/wiki/index.php/Code_Vectorization_Examples

AccelerEyes also plans to put together training materials and tutorials to help programmers learn the art of what we’re calling hardware independent data-parallel programming. The overall idea is that whereas in the past, programmers were required to write low-level code (assembly, CUDA, etc) to realize performance, today the same gains can be made by instead writing at a higher level (vectorized for instance) with the added plus that in doing so, coders aren’t tying their applications down to a particular piece of hardware.  Stay tuned as we will post when these materials become available.

Furthermore, we would encourage you to participate in our User Forum and provide segments of your code where you are trying to improve performance – particularly for-loops.  Our team along with other Jacket users may be able to provide you with some vectorization assistance to help not only your CPU performance but what you can get from the GPU leveraging Jacket.