PAR ECSE-4740-01 Applied Parallel Computing for Engineers, Spring 2018, Rensselaer Polytechnic Institute (Posts about class)

PAR Class 14, Wed 2018-05-02

W Randolph Franklin, RPI — Thu, 26 Apr 2018 04:00:00 GMT

Table of contents

1 Course recap

1 Course recap

My teaching style is to work from particulars to the general.
You've seen OpenMP, a tool for shared memory parallelism.
You've seen the architecture of NVidia's GPU, a widely used parallel system, and CUDA, a tool for programming it.
You've seen Thrust, a tool on top of CUDA, built in the C++ STL style.
You've seen how widely used numerical tools like BLAS and FFT have versions built on CUDA.
You've had a chance to program in all of them on parallel.ecse, with dual 14-core Xeons, Pascal NVidia board, and Xeon Phi coprocessor.
You seen talks by leaders in high performance computing, such as Jack Dongarra.
You've seen quick references to parallel programming using Matlab, Mathematica, and the cloud.
Now, you can inductively reason towards general design rules for shared and non-shared parallel computers, and for the SW tools to exploit them.

PAR Class 13, Wed 2018-04-25

W Randolph Franklin, RPI — Wed, 25 Apr 2018 04:00:00 GMT

Table of contents

1 Inspiration for finishing your term projects

The Underhanded C Contest

"The goal of the contest is to write code that is as readable, clear, innocent and straightforward as possible, and yet it must fail to perform at its apparent function. To be more specific, it should do something subtly evil. Every year, we will propose a challenge to coders to solve a simple data processing problem, but with covert malicious behavior. Examples include miscounting votes, shaving money from financial transactions, or leaking information to an eavesdropper. The main goal, however, is to write source code that easily passes visual inspection by other programmers."
The International Obfuscated C Code Contest
https://www.awesomestories.com/asset/view/Space-Race-American-Rocket-Failures

Moral: After early disasters, sometimes you can eventually get things to work.
The 'Wrong' Brothers Aviation's Failures (1920s)
Early U.S. rocket and space launch failures and explosion
Numerous US Launch Failures

2 Parallel computing videos

We'll see some subset of these.

2.1 Welcome Distributed Systems in One Lesson

niko peikrishvili, 11 min

https://www.youtube.com/watch?v=T9ej3NcE2gQ

This is the first of several short videos.

2.2 Paying for Lunch: C++ in the ManyCore Age

CppCon 2014: by Herb Sutter

https://www.youtube.com/watch?v=AfI_0GzLWQ8

Published on Sep 29, 2014, 1h15m

http://www.cppcon.org

Presentation Slides, PDFs, Source Code and other presenter materials are available at: https://github.com/CppCon/CppCon2014

Concurrency is one of the major focuses of C++17 and one of the biggest challenges facing C++ programmers today. Hear what this panel of experts has to say about how to write concurrent C++ now and in the future.

MODERATOR: Herb Sutter - Author, chair of the ISO C++ committee, software architect at Microsoft.

SPEAKERS:

PABLO HALPERN - Pablo Halpern has been programming in C++ since 1989 and has been a member of the C++ Standards Committee since 2007. He is currently the Parallel Programming Languages Architect at Intel Corp., where he coordinates the efforts of teams working on Cilk Plus, TBB, OpenMP, and other parallelism languages, frameworks, and tools targeted to C++, C, and Fortran users. Pablo came to Intel from Cilk Arts, Inc., which was acquired by Intel in 2009. During his time at Cilk Arts, he co-authored the paper "Reducers and other Cilk++ Hyperobjects", which won best paper at the SPAA 2009 conference. His current work is focused on creating simpler and more powerful parallel programming languages and tools for Intel's customers and promoting adoption of parallel constructs into the C++ and C standards. He lives with his family in southern New Hampshire, USA. When not working on parallel programming, he enjoys studying the viola, skiing, snowboarding, and watching opera. Twitter handle: @PabloGHalpern

JARED HOBEROCK - Jared Hoberock is a research scientist at NVIDIA where he develops the Thrust parallel algorithms library and edits the Technical Specification on Extensions for Parallelism for C++.Website: http://github.com/jaredhoberock

ARTUR LAKSBERG - Artur Laksberg leads the Visual C++ Libraries development team at Microsoft. His interests include concurrency, programming language and library design, and modern C++. Artur is one of the co-authors of the Parallel STL proposal; his team is now working on the prototype implementation of the proposal.

ADE MILLER - Ade Miller writes C++ for fun. He wrote his first N-body model in BASIC on an 8-bit microcomputer 30 years ago and never really looked back. He started using C++ in the early 90s. Recently, he's written two books on parallel programming with C++; "C++ AMP: Accelerated Massive Parallelism with Microsoft Visual C++" and "Parallel Programming with Microsoft Visual C++". Ade spends the long winters in Washington contributing to the open source C++ AMP Algorithms Library and well as a few other projects. His summers are mostly spent crashing expensive bicycles into trees. Website: http://www.ademiller.com/blogs/tech/ Twitter handle: @ademiller

GOR NISHANOV - Gor Nishanov is a is a Principal Software Design Engineer on the Microsoft C++ team. He works on the 'await' feature. Prior to joining C++ team, Gor was working on distributed systems in Windows Clustering team.

MICHAEL WONG - You can talk to me about anything including C++ (even C and that language that shall remain nameless but starts with F), Transactional Memory, Parallel Programming, OpenMP, astrophysics (where my degree came from), tennis (still trying to see if I can play for a living), travel, and the best food (which I am on a permanent quest to eat). Michael Wong is the CEO of OpenMP. He is the IBM and Canadian representative to the C++ Standard and OpenMP Committee. And did I forget to say he is a Director of ISOCPP.org and a VP, Vice-Chair of Programming Languages for Canada's Standard Council. He has so many titles, its a wonder he can get anything done. Oh, and he chairs the WG21 SG5 Transactional Memory, and is the co-author of a number C++11/OpenMP/TM features including generalized attributes, user-defined literals, inheriting constructors, weakly ordered memory models, and explicit conversion operators. Having been the past C++ team lead to IBM´s XL C++ compiler means he has been messing around with designing C++ compilers for twenty years. His current research interest, i.e. what he would like to do if he had time is in the area of parallel programming, transactional memory, C++ benchmark performance, object model, generic programming and template metaprogramming. He holds a B.Sc from University of Toronto, and a Masters in Mathematics from University of Waterloo. He has been asked to speak at ACCU, C++Now, Meeting C++, CASCON, and many Universities, research centers and companies, except his own, where he has to listen. Now he and his wife loves to teach their two children to be curious about everything.

2.3 Combine Lambdas and weak_ptrs to make concurrency easy

https://www.youtube.com/watch?v=fEnnmpdZllQ

CppCon 2016: Dan Higgins, 4min

2.4 A Pragmatic Introduction to Multicore Synchronization

by Samy Al Bahra.

https://www.youtube.com/watch?v=LX4ugnzwggg

Published on Jun 15, 2016, 1h 2m.

This talk will introduce attendees to the challenges involved in achieving high performance multicore synchronization. The tour will begin with fundamental scalability bottlenecks in multicore systems and memory models, and then extend to advanced synchronization techniques involving scalable locking and lock-less synchronization. Expect plenty of hacks and real-world war stories in the fight for vertical scalability. Some of the topics introduced include memory coherence and consistency, memory organization, scalable locking, biased asymmetric synchronization, non-blocking synchronization and safe memory reclamation.

2.5 Synchronization - Blocking & Non-Blocking (1/2)

by Petr Kuznetsov, 15min

https://www.youtube.com/watch?v=k8uOOvd6Uj8

2.6 Lock-Free Programming (or, Juggling Razor Blades), Part I

CppCon 2014, by Herb Sutter

Published on Oct 16, 2014

http://www.cppcon.org

Presentation Slides, PDFs, Source Code and other presenter materials are available at: https://github.com/CppCon/CppCon2014

Example-driven talk on how to design and write lock-free algorithms and data structures using C++ atomic -- something that can look deceptively simple, but contains very deep topics. (Important note: This is not the same as my "atomic Weapons" talk; that talk was about the "what they are and why" of the C++ memory model and atomics, and did not cover how to actually use atomics to implement highly concurrent algorithms and data structures.)

3 Debugging videos

3.1 uftrace: A function graph tracer for C/C++ userspace programs

https://www.youtube.com/watch?v=LNav5qvyK7I

6min

Published on Oct 7, 2016

http://CppCon.org

Presentation Slides, PDFs, Source Code and other presenter materials are available at: https://github.com/cppcon/cppcon2016

This is one of the CppCon 2016 Lightning Talks.

3.2 Concepts in 5

https://www.youtube.com/watch?v=H8HplZtVGT0&list=PLHTh1InhhwT6aWgfHhrvYY3s-lqk0Y9iP&index=14

CppCon 2016: David Sankel

4 Programming videos

4.1 Modern C++: What You Need to Know

https://www.youtube.com/watch?v=TJHgp1ugKGM

by Paulo Portela

Published on Apr 7, 2014, 1h

This talk will give an update on recent progress and near-future directions for C++, both at Microsoft and across the industry. This is a great introduction to the current state of the language, including a glimpse into the future of general purpose, performance-intensive, power-friendly, powerful native programming.

5 Relevant conferences

PAR Class 12, Wed 2018-04-18

W Randolph Franklin, RPI — Wed, 18 Apr 2018 04:00:00 GMT

Table of contents

1 Git
2 Software tips
- 2.1 Freeze decisions early: SW design paradigm
- 2.2 Faster graphical access to parallel.ecse
3 Jack Dongarra videos
4 More parallel tools
- 4.1 cuFFT Notes
- 4.2 cuBLAS etc Notes
- 4.3 Matlab
- 4.4 Mathematica in parallel
5 Nvidia videos
6 Cloud computing

1 Git

Git is good to simultaneously keep various versions. A git intro:

Create a dir for the project:

mkdir PROJECT; cd PROJECT

Initialize:

git init

Create a branch (you can do this several times):

git branch MYBRANCHNAME

Go to a branch:

git checkout MYBRANCHNAME

Do things:

vi, make, ....

Save it:

git add .; git commit -mCOMMENT

Repeat

I might use this to modify a program for class.

2 Software tips

2.1 Freeze decisions early: SW design paradigm

One of my rules is to push design decisions to take effect as early in the process execution as possible. Constructing variables at compile time is best, at function call time is 2nd, and on the heap is worst.

If I have to construct variables on the heap, I construct few and large variables, never many small ones.
Often I compile the max dataset size into the program, which permits constructing the arrays at compile time. Recompiling for a larger dataset is quick (unless you're using CUDA).

Accessing this type of variable uses one less level of pointer than accessing a variable on the heap. I don't know whether this is faster with a good optimizing compiler, but it's probably not slower.
If the data will require a dataset with unpredictably sized components, such as a ragged array, then I may do the following.
1. Read the data once to accumulate the necessary statistics.
2. Construct the required ragged array.
3. Reread the data and populate the array.

2.2 Faster graphical access to parallel.ecse

X over ssh is very slow.

Here are some things I've discovered that help, and that work sometimes.

Use xpra; here's an example:
1. On parallel.ecse:
```
xpra start :77; DISPLAY=:77 xeyes&
```
  Don't everyone use 77, pick your own numbers in the range 20-99.
2. On server, i.e., your machine:
```
xpra attach ssh:parallel.ecse.rpi.edu:77
```
3. I suspect that security is weak. When you start an xpra session, In suspect that anyone on parallel.ecse can display to it. I suspect that anyone with ssh access to parallel.ecse can try to attach to it, and the that 1st person wins.
Use nx, which needs a server, e.g., FreeNX.

3 Jack Dongarra videos

Sunway TaihuLight's strengths and weaknesses highlighted. 9 min. 8/21/2016.

This is the new fastest known machine on top500. A machine with many Intel Xeon Phi coprocessors is now 2nd, Nvidia K20 is 3rd, and some machine built by a company down the river is 4th. These last 3 machines have been at the top for a surprisingly long time.
An Overview of High Performance Computing and Challenges for the Future. 57min, 11/16/2016.

4 More parallel tools

4.1 cuFFT Notes

GPU Computing with CUDA Lecture 8 - CUDA Libraries - CUFFT, PyCUDA from Christopher Cooper, BU
video #8 - CUDA 5.5 cuFFT FFTW API Support. 3 min.
cuFFT is inspired by FFTW (the fastest Fourier transform in the west), which they say is so fast that it's as fast as commercial FFT packages.
I.e., sometimes commercial packages may be worth the money.
Although the FFT is taught for N a power of two, users often want to process other dataset sizes.
The problem is that the optimal recursion method, and the relevant coefficients, depends on the prime factors of N.
FFTW and cuFFT determine the good solution procedure for the particular N.
Since this computation takes time, they store the method in a plan.
You can then apply the plan to many datasets.
If you're going to be processing very many datasets, you can tell FFTW or cuFFT to perform sample timing experiments on your system, to help in devising the best plan.
That's a nice strategy that some other numerical SW uses.
One example is Automatically Tuned Linear Algebra Software (ATLAS).

4.2 cuBLAS etc Notes

BLAS is an API for a set of simple matrix and vector functions, such as multiplying a vector by a matrix.
These functions' efficiency is important since they are the basis for widely used numerical applications.
Indeed you usually don't call BLAS functions directly, but use higher-level packages like LAPACK that use BLAS.
There are many implementations, free and commercial, of BLAS.
cuBLAS is one.
One reason that Fortran is still used is that, in the past, it was easier to write efficient Fortran programs than C or C++ programs for these applications.
There are other, very efficient, C++ numerical packages. (I can list some, if there's interest).
Their efficiency often comes from aggressively using C++ templates.
Matrix mult example

4.3 Matlab

Good for applications that look like matrices.

Considerable contortions required for, e.g., a general graph. You'd represent that with a large sparse adjacency matrix.
Using explicit for loops is slow.
Efficient execution when using builtin matrix functions,

but can be difficult to write your algorithm that way, and

difficult to read the code.
Very expensive and getting more so.

Many separately priced apps.
Uses state-of-the-art numerical algorithms.

E.g., to solve large sparse overdetermined linear systems.

Better than Mathematica.
Most or all such algorithms also freely available as C++ libraries.

However, which library to use?

Complicated calling sequences.

Obscure C++ template error messages.
Graphical output is mediocre.

Mathematica is better.
Various ways Matlab can execute in parallel
1. Operations on arrays can execute in parallel.
  
  E.g. B=SIN(A) where A is a matrix.
2. Automatic multithreading by some functions
  
  Various functions, like INV(a), automatically use perhaps 8 cores.
  
  The '8' is a license limitation.
  
  Which MATLAB functions benefit from multithreaded computation?
3. PARFOR
  
  Like FOR, but multithreaded.
  
  However, FOR is slow.
  
  Many restrictions, e.g., cannot be nested.
  
  Matlab's introduction to parallel solutions
  
  Start pools first with: MATLABPOOL OPEN 12
  
  Limited to 12 threads.
  
  Can do reductions.
4. Parallel Computing Server
  
  This runs on a parallel machine, including Amazon EC2.
  
  Your client sends batch or interactive jobs to it.
  
  Many Matlab toolboxes are not licensed to use it.
  
  This makes it much less useful.
5. GPU computing
  
  Create an array on device with gpuArray
  
  Run builtin functions on it.
  
  Matlab's run built in functions on a gpu

4.4 Mathematica in parallel

You terminate an input command with shift-enter.

Some Mathematica commands:

Sin[1.]
Plot[Sin[x],{x,-2,2}]
a=Import[
 "/opt/parallel/mathematica/mtn1.dat"]
Information[a]
Length[a]
b=ArrayReshape[a,{400,400}]
MatrixPlot[b]
ReliefPlot[b]
ReliefPlot[b,Method->"AspectBasedShading"]
ReliefPlot[MedianFilter[b,1]]
Dimensions[b]
Eigenvalues[b]   *When you get bored*
   * waiting, type * alt-.
Eigenvalues[b+0.0]
Table[ {x^i y^j,x^j y^i},{i,2},{j,2}]
Flatten[Table[ {x^i y^j,x^j y^i},{i,2},{j,2}],1]
StreamPlot[{x*y,x+y},{x,-3,3},{y,-3,3}]
$ProcessorCount
$ProcessorType
*Select *Parallel Kernel Configuration*
   and *Status* in the *Evaluation* menu*
ParallelEvaluate[$ProcessID]
PrimeQ[101]
Parallelize[Table[PrimeQ[n!+1],{n,400,500}]]
merQ[n_]:=PrimeQ[2^n-1]
Select[Range[5000],merQ]
ParallelSum[Sin[x+0.],{x,0,100000000}]
Parallelize[  Select[Range[5000],merQ]]
Needs["CUDALink`"]  *note the back quote*
CUDAInformation[]
Manipulate[n, {n, 1.1, 20.}]
Plot[Sin[x], {x, 1., 20.}]
Manipulate[Plot[Sin[x], {x, 1., n}], {n, 1.1, 20.}]
Integrate[Sin[x]^3, x]
Manipulate[Integrate[Sin[x]^n, x], {n, 0, 20}]
Manipulate[{n, FactorInteger[n]}, {n, 1, 100, 1}]
Manipulate[Plot[Sin[a x] + Sin[b x], {x, 0, 10}],
    {a, 1, 4}, {b, 1, 4}]

Unfortunately there's a problem that I'm still debugging with the Mathematica - CUDA interface.

5 Nvidia videos

HPC and Supercomputing at GTC 2017 1 min
NVIDIA Self-Driving Car Demo at CES 2017 2 min
How Nvidia Went From Gaming to GPUs 3 min
Nvidia Volta To Be Released Q3 2017, Say Rumours | RX 480 Can be Flashed to RX 580 10 min. 4/19/2017
NVIDIA Opening Keynote Highlights at CES 2017 37 min

6 Cloud computing

The material is from Wikipedia, which appeared better than any other sources that I could find.

Hierarchy:
1. IaaS (Infrastructure as a Service)
  1. Sample functionality: VM, storage
  2. Examples:
    1. Google_Compute_Engine
    2. Amazon_Web_Services
    3. OpenStack : compute, storage, networking, dashboard
2. PaaS (Platform ...)
  1. Sample functionality: OS, Web server, database server
  2. Examples:
    1. OpenShift
    2. Cloud_Foundry
    3. Hadoop :
      1. distributed FS, Map Reduce
      2. derived from Google FS, map reduce
      3. used by Facebook etc.
  3. Now, people often run Apache Spark™ - Lightning-Fast Cluster Computing instead of Hadoop, because Spark is faster.
3. SaaS (Software ...)
  1. Sample functionality: email, gaming, CRM, ERP
Cloud_computing_comparison
Virtual machine.

The big question is, at what level does the virtualization occur? Do you duplicate the whole file system and OS, even emulate the HW, or just try to isolate files and processes in the same OS.
1. Virtualization
2. Hypervisor
3. Xen
4. Kernel-based_Virtual_Machine
5. QEMU
6. VMware
7. Containers, docker,
8. Comparison_of_platform_virtual_machines
Distributed storage
See also
1. VNC
2. Grid_computing
  1. decentralized, heterogeneous
  2. used for major projects like protein folding

PAR Class 11, Wed 2018-04-11

W Randolph Franklin, RPI — Thu, 29 Mar 2018 04:00:00 GMT

Table of contents

1 Intel Xeon Phi 7120A

1.1 In general

The Xeon Phi is Intel's brand name for their MIC (for Many Integrated Core Architecture).
The 7120a is Intel's Knights Landing (1st generation) MIC architecure, launched in 2014.
It has 61 cores running about 244 threads clocked at about 1.3GHz.

Having several threads per core helps to overcome latency in fetching stuff.
It has 16GB of memory accessible at 352 GB/s, 30BM L2 cache, and peaks at 1TFlops double precision.
It is a coprocessor on a card accessible from a host CPU on a local network.
It is intended as a supercomputing competitor to Nvidia.
The mic architecture is quite similar to the Xeon.
However executables from one don't run on the other, unless the source was compiled to include both versions in the executable file.
The mic has been tuned to emphasize floating performance at the expense of, e.g., speculative execution.

This helps to make it competitive with Nvidia, even though Nvidia GPUs can have many more cores.
Its OS is busybox, an embedded version of linux.
The SW is called MPSS (Manycore Platform Software Stack).
The mic can be integrated with the host in various ways that I haven't (yet) implemented.
1. Processes on the host can execute subprocesses on the device, as happens with Nvidia CUDA.
2. E.g., OpenMP on the host can run parallel threads on the mic.
3. The mic can page virtual memory from the host.
The fastest machine on top500.org a few years ago used Xeon Phi cards.

The 2nd used Nvidia K20 cards, and the 3rd fastest was an IBM Bluegene.

So, my course lets you use the 2 fastest architectures, and there's another course available at RPI for the 3rd.
Information:

1.2 parallel.ecse's mic

The hostname (of this particular MIC) is parallel-mic0 or mic0.
The local filesystem is in RAM and is reinitialized when mic0 is rebooted.
Parallel:/home and /parallel-class are NFS exported to mic0.
/home can be used to move files back and forth.
All the user accounts on parallel were given accounts on mic0.
You can ssh to mic0 from parallel.
Your current parallel ssh key pair should work.
Your parallel login password as of a few days ago should work on mic0.

However, future changes to your parallel password will not propagate to mic0 and you cannot change your mic0 password.

(The mic0 setup snapshotted parallel's accounts and created a read-only image to boot mic0 from. Any changes to mic0:/etc/shadow are reverted when mic0 reboots.)

So use your public key.

1.3 Programming the mic

Parallel:/parallel-class/mic/bin has versions of gcc, g++, etc, with names like k1om-mpss-linux-g++ .
These run on parallel and produce executable files that run (only) on mic0.
Here's an example of compiling (on parallel) a C program in /parallel-class/mic
```
bin/k1om-mpss-linux-gcc hello.c -o hello-mic
```
Run it thus from parallel (it runs on mic0):
```
ssh mic0  /parallel-class/mic/hello-mic
```

2 Intel compilers on parallel

Note: currently they don't work, but maybe soon..

Intel Parallel Studio XE Cluster 2017 is now installed on parallel, in /opt/intel/ . It is a large package with compilers, debuggers, analyzers, MPI, etc, etc. There is is extensive doc on Intel's web site. Have fun.

Students and free SW developers can get also free licenses for their machines. Commercial licenses cost $thousands.
What’s Inside Intel Parallel Studio XE 2017. There're PDF slides, a webinar, and training stuff.
https://goparallel.sourceforge.net/want-faster-code-faster-check-parallel-studio-xe/
Before using the compiler, you should initialize some envars thus:
```
source /opt/intel/bin/iccvars.sh -arch intel64
```

Then you can compile a C or C++ program thus:

icc -openmp -O3 foo.c -o foo
icpc -qopenmp -O3 foo.cc -o foo

On my simple tests, not using the mic, icpc and g++ produced equally good code.

Compile a C++ program with OpenMP thus:

icpc -qopenmp -std=c++11    omp_hello.cc   -o omp_hello

Test it thus, it is in /parallel-class/mic
```
OMP_NUM_THREADS=4 ./omp_hello
```
Note how the output from the various threads is mixed up.

3 Programming the MIC (ctd)

It turns out that I (but not you) can update a login password on mic0, but it's a little tedious. Use your ssh key.

Details: at startup, mic0:/etc is initialized from parallel:/var/mpss/mic0/etc So I could edit shadow and insert a new encrypted password.

So is /home, but it's then replaced by the NFS mount.
I can also change, e.g., your login shell. Use bash on mic0 since zsh does not exist there.
Dr.Dobb's: Programming the Xeon Phi by Rob Farber.
Some MIC demos .
To cross compile with icc and icpc, see the MPSS users guide, Section 8.1.4. Use the -mmic flag.
Intel OpenMP* Support Overview.
New Era for OpenMP*: Beyond Traditional Shared Memory Parallel Programming.
Book: Parallel Programming and Optimization with Intel® Xeon Phi™ Coprocessors, 2nd Edition [508 Pages].
See also https://www.cilkplus.org/
Rolling Out the New Intel Xeon Phi Processor at ISC 2016 https://www.youtube.com/watch?v=HDPYymREyV8
Supermicro Showcases Intel Xeon Phi and Nvidia P100 Solutions at ISC 2016 https://www.youtube.com/watch?v=nVWqSjt6hX4
Insider Look: New Intel® Xeon Phi™ processor on the Cray® XC™ Supercomputer https://www.youtube.com/watch?v=lkf3U_5QG_4

4 Mic0 (Xeon Phi) setup

mic0 is the hostname for the Xeon Phi coprocessor.
1. It's logically a separate computer on a local net named mic0, which is established when the mpss service starts.
2. It's accessible only from parallel.
3. It has its own filesystem, accounts, etc.
On parallel, /opt and /home are exported by being listed in /etc/exports
mic0 has no disk; its root partition is in memory.
parallel:/var/mpss/mic0/ is copied as mic0's root partition.
1. This is copied when the mic boots.
2. That is 1-way; changes done on mic0 are not copied back.
3. Changes to parallel:/var/mpss/mic0/ are not visible to the mic until it reboots.
Accounts on parallel have home dirs on mic0.
I add the accounts themselves by copying the relevant lines from parallel:/etc/{passwd,shadow,group} to /var/mpss/mic0/etc/{passwd,shadow,group}.
parallel is running an old linux kernel, 4.4.55, because the required kernel module, mic.ko, wouldn't compile with a newer kernel, even 4.8.
1. The proximate problem is that newer kernels have secure boot, where modules need to be validated.
2. This can allegedly be disabled, but that didn't work.
3. I also tried to create a certificate authorizing mic.ko, but that didn't work.
4. It's possible that sufficient work might get a newer kernel to work. However I spent far too much time getting it to this point.
FYI, parallel:/opt/mpss/ has some relevant files.
parallel:/opt/intel has useful Intel compilers and tools, most of which I haven't gotten working yet. If anyone wants to do this, welcome.
On parallel, micctrl -s gives the mic's state.
micctrl has other rooty options.
On parallel, root controls the mic with service mpss start/stop
The mpss service should start automatically when parallel reboots.

5 Boinc

I've installed BOINC on parallel.ecse.
Currently, it's running MilkyWay@Home.
The command boincmgr gives its status.
Would you like to play with it?
If you want to run timing tests for other SW on parallel, tell me; I'll disable it.

6 Quantum physics talk at 4pm today

The Fascinating Quantum World of Two-dimensional Materials: Symmetry, Interaction and Topological Effects

Symmetry, interaction and topological effects, as well as environmental screening, dominate many of the quantum properties of reduced-dimensional systems and nanostructures. These effects often lead to manifestation of counter-intuitive concepts and phenomena that may not be so prominent or have not been seen in bulk materials. In this talk, I present some fascinating physical phenomena discovered in recent studies of atomically thin two-dimensional (2D) materials. A number of highly interesting and unexpected behaviors have been found – e.g., strongly bound excitons (electron-hole pairs) with unusual energy level structures and new topology-dictated optical selection rules, massless excitons, tunable magnetism and plasmonic properties, electron supercollimation, novel topological phases, etc. – adding to the promise of these 2D materials for exploration of new science and valuable applications.

Steven G. Louie, Physics Department, University of California at Berkeley, and Lawrence Berkeley National Lab

Darrin Communications Center (DCC) 337 4:00 pm

Announcement (link will decay soon.)

PAR Class 10, Wed 2018-03-28

W Randolph Franklin, RPI — Tue, 27 Mar 2018 04:00:00 GMT

Table of contents

1 /parallel-class
2 OpenMP: CPU time vs wall clock (elapsed) time
3 More Thrust info
4 Linux programming tip
5 Thrust
- 5.1 Examples
- 5.2 Backends
6 Intel compilers on parallel
7 Programming the MIC (ctd)
8 OpenMP on the mic

1 /parallel-class

This dir on parallel.ecse has some of my modified programs that are not on geoxeon.ecse, such as tiled_range2.cu (mentioned below).

2 OpenMP: CPU time vs wall clock (elapsed) time

On parellel.ecse, /parallel-class/openmp/rpi/cpuvs.wall.cc is a simple program that shows how wallclock time shrinks as the number of threads rises (up to a point) but CPU time rises from the start.

That is because one way that a thread waits to run is to burn cycles. For small delays, this is easier and has less overhead than explicitly suspending. The GNU version of OpenMP lets you choose the crossover point between buring cycles and syspending a thread.

3 More Thrust info

/parallel-class/thrust/doc/An_Introduction_To_Thrust.pdf
GTC_2010_Part_2_Thrust_By_Example.pdf

4 Linux programming tip

I like to use emacs on my local machine to edit files on parallel.ecse. It's more responsive than running a remote editor, and lets me copy to and from local files. Here are 2 ways to do that.

In emacs on my local machine, I do find-file thus:

/parallel.ecse.rpi.edu:/parallel-class/thrust/rpi/tiled_range2.cu

Emacs transparently runs scp to read and write the remote file.
I can mount remote directories on my local machine with something like ssh-fs and then edit remote files as if they were local.

In either case, I have to compile the files on parallel.

5 Thrust

5.1 Examples

I rewrote /parallel-class/thrust/examples-1.8/tiled_range.cu into /parallel-class/thrust/rpi/tiled_range2.cu .

It is now much shorter and much clearer. All the work is done here:

gather(make_transform_iterator(make_counting_iterator(0), _1%N), make_transform_iterator(make_counting_iterator(N*C), _1%N), data.begin(), V.begin());
1. make_counting_iterator(0) returns pointers to the sequence 0, 1, 2, ...
2. _1%N is a function computing modulo N.
3. make_transform_iterator(make_counting_iterator(0), _1%N) returns pointers to the sequence 0%N, 1%N, ...
4. gather populates V. The i-th element of V gets make_transform_iterator...+i element of data, i.e., the i%N-th element of data.
tiled_range3.cu is even shorter. Instead of writing an output vector, it constructs an iterator for a virtual output vector:

auto output=make_permutation_iterator(data, make_transform_iterator(make_counting_iterator(0), _1%N));
1. *(output+i) is *(data+(i%N)).
2. You can get as many tiles as you want by iterating.
3. tiled_range3.cu also constructs an iterator for a virtual input vector (in this case a vector of squares) instead of storing the data:
auto data = make_transform_iterator(make_counting_iterator(0), _1*_1);
tiled_range5.cu shows how to use a lambda instead of the _1 notation:

auto output=make_permutation_iterator(data, make_transform_iterator(make_counting_iterator(0), [](const int i){return i%N;} ));
1. You have to compile with --std c++11 .
2. This can be rewritten thus:
  
  auto f = [](const int i){return i%N;}; auto output = make_permutation_iterator(data, make_transform_iterator(make_counting_iterator(0), f));
3. The shortest lambda is this:
  
  auto f = [](){};
repeated_range2.cu is my improvement on repeated_range.cu:

auto output=make_permutation_iterator(data.begin(), make_transform_iterator(make_counting_iterator(0), _1/3));
1. make_transform_iterator(make_counting_iterator(0), _1/3)) returns pointers to the sequence 0,0,0,1,1,1,2,2,2, ...
Unmodified thrust examples:
1. expand.cu takes a vector like V= [0, 10, 20, 30, 40] and a vector of repetition counts, like C= [2, 1, 0, 3, 1]. Expand repeats each element of V the appropriate number of times, giving [0, 0, 10, 30, 30, 30, 40]. The process is as follows.
  1. Since the output vector will be longer than the input, the main program computes the output size, by reduce summing C, and constructs a vector to hold the output.
  2. Exclusive_scan C to obtain output offsets for each input element: C2 = [0, 2, 3, 3, 6].
  3. Scatter_if the nonzero counts into their corresponding output positions. A counting iterator, [0, 1, 2, 3, 4] is mapped with C2, using C as the stencil, giving C3 = [0, 0, 1, 3, 0, 0, 4].
  4. An inclusive_scan with max fills in the holes in C3, to give C4 = [0, 0, 1, 3, 3, 3, 4].
  5. Gather uses C4 to gather elements of V: [0, 0, 10, 30, 30, 30, 40].
2. set_operations.cu. This shows methods of handling an operation whose output is of unpredictable size. The question is, is space or time more important?
  1. If the maximum possible output size is reasonable, then construct an output vector of that size, use it, and then erase it down to its actual size.
  2. Or, run the operation twice. The 1st time, write to a discard_iterator, and remember only the size of the written data. Then, construct an output vector of exactly the right size, and run the operation again.
    
    I use this technique a lot with ragged arrays in sequential programs.
3. sparse_vector.cu represents and sums sparse vectors.
  1. A sparse vector has mostly 0s.
  2. The representation is a vector of element indices and another vector of values.
  3. Adding two sparse vectors goes as follows.
    1. Allocate temporary index and element vectors of the max possible size (the sum of the sizes of the two inputs).
    2. Catenate the input vectors.
    3. Sort by index.
    4. Find the number of unique indices by applying inner_product with addition and not-equal-to-next-element to the indices, then adding one.
      
      E.g., applied to these indices: [0, 3, 3, 4, 5, 5, 5, 8], it gives 5.
    5. Allocate exactly enough space for the output.
    6. Apply reduce_by_key to the indices and elements to add elements with the same keys.
      
      The size of the output is the number of unique keys.
What's the best way to sort 16000 sets of 1000 numbers each? E.g., sort the rows of a 16000x1000 array? On geoxeon, /pc/thrust/rpi/comparesorts.cu, which I copied from http://stackoverflow.com/questions/28150098/how-to-use-thrust-to-sort-the-rows-of-a-matrix|stackoverflow, compares three methods.
Call the thrust sort 16000 times, once per set. That took 10 secs.
Sort the whole list of 16,000,000 numbers together. Then sort it again by key, with the keys being the set number, to bring the elements of each set together. Since the sort is stable, this maintains the order within each set. (This is also how radix sort works.) That took 0.04 secs.
Call a thrust function (to sort a set) within another thrust function (that applies to each set). This is new in Thrust 1.8. That took 0.3 secs.

This is a surprising and useful paradigm. It works because

There's an overhead to starting each thrust function, and
Radix sort, which thrust uses for ints, takes linear time.

5.2 Backends

The Thrust device can be CUDA, OpenMP, TBB, etc.
You can spec it in 2 ways:
1. by adding an extra arg at the start of a function's arg list.
2. with an envar
  
  https://github.com/thrust/thrust/wiki/Host-Backends
  
  https://github.com/thrust/thrust/wiki/Device-Backends

6 Intel compilers on parallel

Intel Parallel Studio XE Cluster 2017 is now installed on parallel, in /opt/intel/ . It is a large package with compilers, debuggers, analyzers, MPI, etc, etc. There is is extensive doc on Intel's web site. Have fun.

Students and free SW developers can get also free licenses for their machines. Commercial licenses cost $thousands.
What’s Inside Intel Parallel Studio XE 2017. There're PDF slides, a webinar, and training stuff.
https://goparallel.sourceforge.net/want-faster-code-faster-check-parallel-studio-xe/
Before using the compiler, you should initialize some envars thus:
```
source /opt/intel/bin/iccvars.sh -arch intel64
```

Then you can compile a C or C++ program thus:

icc -openmp -O3 foo.c -o foo
icpc -qopenmp -O3 foo.cc -o foo

On my simple tests, not using the mic, icpc and g++ produced equally good code.

Compile a C++ program with OpenMP thus:

icpc -qopenmp -std=c++11    omp_hello.cc   -o omp_hello

Test it thus, it is in /parallel-class/mic
```
OMP_NUM_THREADS=4 ./omp_hello
```
Note how the output from the various threads is mixed up.

7 Programming the MIC (ctd)

Dr.Dobb's: Programming the Xeon Phi by Rob Farber.
Some MIC demos .
To cross compile with icc and icpc, see the MPSS users guide, Section 8.1.4. Use the -mmic flag.
Intel OpenMP* Support Overview.
New Era for OpenMP*: Beyond Traditional Shared Memory Parallel Programming.
Book: Parallel Programming and Optimization with Intel® Xeon Phi™ Coprocessors, 2nd Edition [508 Pages].
See also https://www.cilkplus.org/
Rolling Out the New Intel Xeon Phi Processor at ISC 2016 https://www.youtube.com/watch?v=HDPYymREyV8
Supermicro Showcases Intel Xeon Phi and Nvidia P100 Solutions at ISC 2016 https://www.youtube.com/watch?v=nVWqSjt6hX4
Insider Look: New Intel® Xeon Phi™ processor on the Cray® XC™ Supercomputer https://www.youtube.com/watch?v=lkf3U_5QG_4

8 OpenMP on the mic

OpenMP is now running on the mic (Xeon Phi).

Setup envars thus (assuming you're using bash or zsh):

source /opt/intel/bin/iccvars.sh arch intel64
export SINK_LD_LIBRARY_PATH=/opt/intel/lib/mic

To compile on parallel for running on parallel:
```
icpc -fopenmp sum_reduc2.cc -o sum_reduc2
```
Run it on parallel thus:
```
./sum_reduc2
```

To cross compile on parallel for running on mic0:

icpc -mmic -fopenmp sum_reduc2.cc -o sum_reduc2-mic

Run it natively on mic0 thus:
```
micnativeloadex sum_reduc2-mic
```
It is also possible to have an OpenMP program on parallel.ecse execute parallel threads on mic0:
```
#pragma offload target (mic)
  {
  #pragma omp parallel
    {
    ...
    }
  }
```
See /parallel-class/mic/stackoverflow/hello.c
Compile it thus:
```
icc -fopenmp hello.c -o hello
```
Note that there is no -mmic.
Run it thus:
```
./hello
```
/parallel-class/mic/stackoverflow/hello_omp.c shows a different syntax.
Degrees of parallelism: See slide 12 of https://software.intel.com/sites/default/files/managed/87/c3/using-nested-parallelism-in-openMP-r1.pdf

PAR Class 9, Wed 2018-03-21

W Randolph Franklin, RPI — Wed, 21 Mar 2018 04:00:00 GMT

Table of contents

1 parallel.ecse hardware details

I put the invoice on parallel.ecse in /parallel-class/ . It gives the hardware specifics.

2 Nvidia GPU summary

Here's a summary of the Nvidia Pascal GP104 GPU architecture as I understand it. It's more compact than I've found elsewhere. I'll add to it from time to time. Some numbers are probably wrong.

The host is the CPU.
The device is the GPU.
The device contains 20 streaming multiprocessors (SMs).

Different GPU generations have used the terms SMX or SMM.
A thread is a sequential program with private and shared memory, program counter, etc.
Threads are grouped, 32 at a time, into warps.
Warps of threads are grouped into blocks.

Often the warps are only implicit, and we consider that the threads are grouped directly into blocks.

That abstract hides details that may be important; see below.
Blocks of threads are grouped into a grid, which is all the threads in the kernel.
A kernel is a parallel program executing on the device.
1. The kernel runs potentially thousands of threads.
2. A kernel can create other kernels and wait for their completion.
3. There may be a limit, e.g., 5 seconds, on a kernel's run time.
Thread-level resources:
1. Each thread can use up to 255 fast registers. Registers are private to the thread.
  
  All the threads in one block have their registers allocated from a fixed pool of 65536 registers. The more registers that each thread uses, the fewer warps in the block can run simultaneously.
2. Each thread has 512KB slow local memory, allocated from the global memory.
3. Local memory is used when not enough registers are available, and to store thread-local arrays.
Warp-level resources:
1. Threads are grouped, 32 at a time, into warps.
2. Each warp executes as a SIMD, with one instruction register. At each cycle, every thread in a warp is either executing the same instruction, or is disabled. If the 32 threads want to execute 32 different instructions, then they will execute one after the other, sequentially.
  
  If you read in some NVidia doc that threads in a warp run independently, then continue reading the next page to get the info mentioned in the previous paragraph.
3. If successive instructions in a warp do not depend on each other, then, if there are enough warp schedulers available, they may be executed in parallel. This is called Instruction Level Parallelism (ILP).
4. For an array in local memory, which means that each thread will have its private copy, the elements for all the threads in a warp are interleaved to potentially increase the I/O rate.
  
  Therefore your program should try to have successive threads read successive words of arrays.
5. A thread can read variables from other threads in the same warp, with the shuffle instruction. Typical operation are to read from the K-th next thread, to do a butterfly permutation, or to do an indexed read. This happens in parallel for the whole warp, and does not use shared memory.
6. A warp vote combines a bit computed by each thread to report results like all or any.
Block-level resources:
1. A block may contain up to 1024 threads.
2. Each block has access to 65536 fast 32-bit registers, for the use of its threads.
3. Each block can use up to 49152 bytes of the SM's fast shared memory. The block's shared memory is shared by all the threads in the block, but is hidden from other blocks.
  
  Shared memory is basically a user-controllable cache of some global data. The saving comes from reusing that shared data several times after you loaded it from global memory once.
  
  Shared memory is interleaved in banks so that some access patterns are faster than others.
4. Warps in a block run asynchronously and run different instructions. They are scheduled and executed as resources are available.
5. The threads in a block can be synchonized with __syncthreads().
  
  Because of how warps are scheduled, that can be slow.
6. The threads in a block can be arranged into a 3D array, up to 1024x1024x64.
  
  That is for convenience, and does not increase performance (I think).
7. I'll talk about textures later.
Streaming Multiprocessor (SM) - level resources:
1. Each SM has 128 single-precision CUDA cores, 64 double-precision units, 32 special function units, and 32 load/store units.
2. In total, the GPU has 2560 CUDA cores.
3. A CUDA core is akin to an ALU. The cores, and all the units, are pipelined.
4. A CUDA core is much less powerful than one core of an Intel Xeon. My guess is 1/20th.
5. Beware that, in the CUDA C Programming Guide, NVidia sometimes calls an SM a core.
6. The limited number of, e.g., double precision units means that an DP instruction will need to be scheduled several times for all the threads to execute it. That's why DP is slower.
7. Each SM has 4 warp schedulers and 8 instruction dispatch units.
8. 64 warps can simultaneously reside in an SM.
9. Therefore up to 32x64=2048 threads can be executed in parallel by an SM.
10. Up to 16 blocks that can simultaneously be resident in an SM.
  
  However, if each block uses too many resources, like shared memory, then this number is reduced.
  
  Each block sits on only one SM; no block is split. However a block's warps are executed asynchronously (until synced).
11. Each SM has 64KiB (?) fast memory to be divided between shared memory and an L1 cache. Typically, 48KiB (96?) is used for the shared memory, to be divided among its resident blocks, but that can be changed.
12. The 48KB L1 cache can cache local or global memory.
13. Each SM has a read-only data cache of 48KB to cache the global constant memory.
14. Each SM has 8 texture units, and many other graphics capabilities.
15. Each SM has 256KB of L2 cacha.
Grid-level resources:
1. The blocks in a grid can be arranged into a 3D array. up to $(2^{31}-1, 2^{16}-1, 2^{16}-1)$.
2. Blocks in a grid might run on different SMs.
3. Blocks in a grid are queued and executed as resources are available, in an unpredictable parallel or serial order. Therefore they should be independent of each other.
4. The number of instructions in a kernel is limited.
5. Any thread can stop the kernel by calling assert.
Device-level resources:
1. There is a large and slow 8GB global memory, which persists from kernel to kernel.
  
  Transactions to global memory are 128 bytes.
  
  Host memory can also be memory-mapped into global memory, although the I/O rate will be lower.
  
  Reading from global memory can take hundreds of cycles. A warp that does this will be paused and another warp started. Such context switching is very efficient. Therefore device throughput stays high, although there is a latency. This is called Thread Level Parallelism (TLP) and is a major reason for GPU performance.
  
  That assumes that an SM has enough active warps that there is always another warp available for execution. That is a reason for having warps that do not use all the resources (registers etc) that they're allowed to.
2. There is a 2MB L2 cache, for sharing data between SMs.
3. There is a 64KiB Small and fast global constant memory, , which also persists from kernel to kernel. It is implemented as a piece of the global memory, made fast with caches.
  
  (Again, I'm still resolving this apparent contradiction).
4. Grid Management Unit (GMU) schedules (pauses, executes, etc) grids on the device. This is more important because grids can start other grids (Dynamic Parallelism).
5. Hyper-Q: 32 simultaneous CPU tasks can launch kernels into the queue; they don't block each other. If one kernel is waiting, another runs.
6. CUDA Work Distributor (CWD) dispatches 32 active grids at a time to the SMs. There may be 1000s of grids queued and waiting.
7. GPU Direct: Other devices can DMA the GPU memory.
8. The base clock is 1607MHz.
9. GFLOPS: 8873.
10. Memory bandwidth: 320GB/s
GPU-level resources:
1. Being a Geforce product, there are many graphics facilities that we're not using.
2. There are 4 Graphics processing clusters (GPCs) to do graphics stuff.
3. Several perspective projections can be computed in parallel, for systems with several displays.
4. There's HW for texture processing.
Generational changes:
1. With each new version, Nvidia tweaks the numbers. Some get higher, others get lower.
  1. E.g., Maxwell had little HW for double precision, and so that was slow.
  2. Pascal's clock speed is much higher.
Refs:
1. The CUDA program deviceDrv.
2. http://developer.download.nvidia.com/compute/cuda/compute-docs/cuda-performance-report.pdf
3. http://international.download.nvidia.com/geforce-com/international/pdfs/GeForce_GTX_1080_Whitepaper_FINAL.pdf
4. Better Performance at Lower Occupancy, Vasily Volkov, UC Berkeley, 2010.
5. https://www.pgroup.com/lit/articles/insider/v2n1a5.htm - well written but old.
(I'll keep adding to this. Suggestions are welcome.)

3 More CUDA

CUDA function qualifiers:
1. __global__ device function called from host, starting a kernel.
2. __device__ device function called from device function.
3. __host__ (default) host function called from host function.
CUDA variable qualifiers:
1. __shared__
2. __device__ global
3. __device__ __managed__ automatically paged between host and device.
4. __constant__
5. (nothing) register if scalar, or local if array or if no more registers available.
If installing CUDA on your machine, this repository seems best:

http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64

That includes the Thrust headers but not example programs.

4 Thrust

Thrust is an API that looks like STL. Its backend can be CUDA, OpenMP, or sequential host-based code.
The online Thrust directory structure is a mess. Three main sites appear to be these:
1. https://github.com/thrust -
  1. The best way to install it is to clone from here.
  2. The latest version of the examples is also here.
  3. The wiki has a lot of doc.
2. https://thrust.github.io/
  
  This points to the above site.
3. https://developer.nvidia.com/thrust
  
  This has links to other Nvidia docs, some of which are obsolete.
4. http://docs.nvidia.com/cuda/thrust/index.html
  
  easy-to-read, thorough, obsolete, doc
5. https://code.google.com/ - no longer exists.
The latest version is 1.8.3.
Functional-programming philosophy.
Many possible backends: host, GPU, OpenMP, TBB...
Easier programming, once you get used to it.
Code is efficient.
Uses some unusual C++ techniques, like overloading operator().
Since the Stanford slides were created, Thrust has adopted unified addressing, so that pointers know whether they are host or device.
On parallel in /parallel-class/thrust/ are many little demo programs from the thrust distribution, with my additions.
CUDACast videos on Thrust:

CUDACast #.15 - Introduction to Thrust

CUDACast #.16 - Thrust Algorithms and Custom Operators
Thrust is fast because the functions that look like they would need linear time really take only log time in parallel.
In functions like reduce and transform, you often see an argument like thrust::multiplies<float>(). The syntax is as follows:
1. thrust::multiplies<float> is a class.
2. It overloads operator().
3. However, in the call to reduce, thrust::multiplies<float>() is calling the default constructor to construct a variable of class thrust::multiplies<float>, and passing it to reduce.
4. reduce will treat its argument as a function name and call it with an argument, triggering operator().
5. You may also create your own variable of that class, e.g., thrust::multiplies<float> foo. Then you just say foo in the argument list, not foo().
6. The optimizing compiler will replace the operator() function call with the defining expression and then continue optimizing. So, there is no overhead, unlike if you passed in a pointer to a function.
Sometimes, e.g., in saxpy.cu, you see saxpy_functor(A).
1. The class saxpy_functor has a constructor taking one argument.
2. saxpy_functor(A) constructs and returns a variable of class saxpy_functor and stores A in the variable.
3. The class also overloads operator().
4. (Let's call the new variable foo). foo() calls operator() for foo; its execution uses the stored A.
5. Effectively, we did a closure of saxpy_functor; this is, we bound a property and returned a new, more restricted, variable or class.
The Thrust examples teach several non-intuitive paradigms. As I figure them out, I'll describe a few. My descriptions are modified and expanded versions of the comments in the programs. This is not a list of all the useful programs, but only of some where I am adding to their comments.
1. arbitrary_transformation.cu and dot_products_with_zip.cu. show the very useful zip_iterator. Using it is a 2-step process.
  1. Combine the separate iterators into a tuple.
  2. Construct a zip iterator from the tuple.
  Note that operator() is now a template.
2. boundingbox.cu finds the bounding box around a set of 2D points.
  
  The main idea is to do a reduce. However, the combining operation, instead of addition, is to combine two bounding boxes to find the box around them.
3. bucket_sort2d.cu overlays a grid on a set of 2D points and finds the points in each grid cell (bucket).
  1. The tuple is an efficient class for a short vector of fixed length.
  2. Note how random numbers are generated. You combine an engine that produces random output with a distribution.
    
    However you might need more complicated coding to make the numbers good when executing in parallel. See monte_carlo_disjoint_sequences.cu.
  3. The problem is that the number of points in each cell is unpredictable.
  4. The cell containing each point is computed and that and the points are sorted to bring together the points in each cell.
  5. Then lower_bound and upper_bound are used to find each bucket in that sorted vector of points.
  6. See this lower_bound description.
4. mode.cu shows:
  1. Counting the number of unique keys in a vector.
    1. Sort the vector.
    2. Do an inner_product. However, instead of the operators being times and plus, they are not equal to the next element and plus.
  2. Counting their multiplicity.
    1. Construct vectors, sized at the number of unique keys, to hold the unique keys and counts.
    2. Do a reduce_by_keys on a constant_iterator using the sorted vector as the keys. For each range of identical keys, it sums the constant_iterator. That is, it counts the number of identical keys.
    3. Write a vector of unique keys and a vector of the counts.
  3. Finding the most used key (the mode).
    1. Do max_element on the counts vector.
5. repeated_range.cu repeats each element of an N-vector K times: repeated_range([0, 1, 2, 3], 2) -> [0, 0, 1, 1, 2, 2, 3, 3]. It's a lite version of expand.cu, but uses a different technique.
  1. Here, N=4 and K=2.
  2. The idea is to construct a new iterator, repeated_range, that, when read and incremented, will return the proper output elements.
  3. The construction stores the relevant info in structure components of the variable.
  4. Treating its value like a subscript in the range [0,N*K), it divides that value by K and returns that element of its input.
  See also strided_range.cu and tiled_range.cu.

5 Unionfs: Linux trick of the day

aka overlay FS, translucent FS.
If a, b are directories, and m is an empty directory, then

unionfs -o cow a=RW:b m

makes m to be a combo of a and b, with a being higher priority
Writing a file into m writes it in a.
Changing a file in b writes the new version into a
Deleting a file in b causes a white-out note to be stored in a.
Unmount it thus:

fusermount -u m
None of this requires superuser.
Application: making a read-only directory into a read-write directory.
Note: IBM had a commercial version of this idea in its CP/CMS OS in the 1960s.

PAR Class 8, Wed 2018-03-07

W Randolph Franklin, RPI — Wed, 07 Mar 2018 05:00:00 GMT

Class cancelled because of weather.

PAR Class 7, Wed 2018-02-28

W Randolph Franklin, RPI — Tue, 27 Feb 2018 05:00:00 GMT

Table of contents

1 Term project progress

How's this going? Send me a progress report.

2 Parallel programs

/parallel-class/cuda/matmul2.cu plays with matrix multiplication of two random 1000x1000 matrices, and with managed memory.

It shows a really quick way to use OpenMP, which has a 10x speedup.
It shows a really quick way to use CUDA, which has a 15x speedup. This just uses one thread block per input matrix A row, and one thread per output element. That's 1M threads. The data is read from managed memory. Note how easy it is. There is no storing tiles into fast shared memory.
It multiplies the matrices on the host, and compares reading them from normal memory and from managed memory. The latter is 2.5x slower. Dunno why.
matmul2 also shows some of my utility functions and macros.
1. InitCUDA prints a description of the GPU etc.
2. cout << PRINTC(expr) prints an expression's name and then its value and a comma.
3. PRINTN is like PRINTC but ends with endl.
4. TIME(expr) evals an expression then prints its name and total and delta
  1. CPU time,
  2. elapsed time,
  3. their ratio.
5. CT evals and prints the elapsed time of a CUDA kernel.
6. Later I may add new tests to this.
/parallel-class/cuda/checksum.cc shows a significant digits problem when you add many small numbers.
sum_reduction.cu is Stanford's program.
sum_reduction2.cu is my modification to use managed memory.

Note how both sum_reduction and sum_reduction2 give different answers for the serial and the parallel computation. That is bad.
sum_reduction3.cu is a mod to try to find the problem. One problem is insufficient precision in the sum. Using double works. However there might be other problems.

3 Computer factoid

Unrelated to this course, but perhaps interesting:

All compute servers have for decades had a microprocessor frontend that controls the boot process. The current iteration is called an IPMI. It has a separate ethernet port, and would allow remote bios configs and booting. On parallel, that port is not connected (I don't trust the security).

4 CUDA Doc

The start of Nvidia's Parallel thread execution has useful info.

This is one of a batch of CUDA docs. Browse as you wish.

5 Stanford lectures

All the lectures and associated files are on geoxeon and parallel.ecse at /parallel-class/stanford/ .

They are also online here.

Lecture 6 parallel patterns 1 presents some paradigms of parallel programming. These are generally useful building blocks for parallel algorithms.

6 Thrust

Stanford lecture 8: Thrust. This is a functional frontend to various backends, such as CUDA.

Programs are in /parallel-class/thrust/ .

7 IBM's quantum computer

As mentioned by Narayanaswami Chandrasekhar last week.

https://en.wikipedia.org/wiki/IBM_Quantum_Experience
https://techcrunch.com/2017/11/10/ibm-passes-major-milestone-with-20-and-50-qubit-quantum-computers-as-a-service/
https://www.technologyreview.com/s/609451/ibm-raises-the-bar-with-a-50-qubit-quantum-computer/
https://www.research.ibm.com/ibm-q/ - points to lots of info, e.g., QISKit on github, beginners guide, etc.

You to do: learn this and present it to me in class.

PAR Class 6, Wed 2018-02-21

W Randolph Franklin, RPI — Thu, 15 Feb 2018 05:00:00 GMT

Table of contents

1 Narayanaswami Chandrasekhar talk on Blockchains

Class will be short today because of this.

2 Optional Homework - bring your answers and discuss next week

2.1 Paper questions

Research and then describe the main changes from NVidia Maxwell to Pascal.
Although a thread can use 255 registers, that might be bad for performance. Why?
Give a common way that the various threads in a block can share data with each other.
Reading a word from global memory might take 400 cycles. Does that mean that a thread that reads many words from global memory will always take hundreds of times longer to complete?
Since the threads in a warp are executed in a SIMD fashion, how can an if-then-else block be executed?
What is unified virtual addressing and how does it make CUDA programming easier?

2.2 Programming questions

Repeat homework 2's matrix multiplication problem, this time in CUDA. Report how much parallel speedup you get.
Look at the dataset /parallel-class/data/bunny. It contains 35947 points for the Stanford bunny.

Assuming that each point has a mass of 1, and is gravitationally attracted to the others, compute the potential energy of the system. The formula is this:

$U = - \sum_{i=1}^{N-1} \sum_{j=i+1}^N \frac{1}{r_{ij}}$

where $r_{ij}$ is the distance between points $i$ and $j$ . (This assumes that G=1).
Now look at the dataset /parallel-class/data/blade, which contains 882954 points for a turbine blade. Can you process it?

3 Stanford lectures

Lecture 5 performance considerations shows how to fine tune your program once it's already working, if you need the extra speed.
Lecture 6 parallel patterns 1 presents some paradigms of parallel programming. These are generally useful building blocks for parallel algorithms.

4 Misc CUDA

The demo programs are in /local/cuda/samples/ . Their coding style is suboptimal. However, in /local/cuda/samples/1_Utilities/ , bandwidthTest and deviceQuery are interesting.

For your convenience, /parallel-class/deviceQuery is a link. Run it to see the GPU's capabilities.
The program nvidia-smi shows the current load on the GPU.
My web copy of the tutorial programs from Stanford's parallel course notes is also on parallel at /parallel-class/stanford/tutorials/ .
1. I've edited some of them, and put the originals in orig/ , and created new ones.
2. To compile them, you need /local/cuda/bin in your PATH and /local/cuda/lib64 in your LD_LIBRARY_PATH .
3. Name your source program foo.cu for some foo .
4. Compile it thus: nvcc foo.cu -o foo .
5. hello_world.cu shows a simple CUDA program and uses a hack to print from a device function.
6. hello_world2.cu shows printing from several threads.
7. global_functions.cu shows some basic CUDA stuff.
8. device_functions.cu extends it.
9. vector_addition.cu does (you figure it out).
10. vector_addition2.cu is my modification to use unified memory, per http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html . I also cleaned up the code and shrank the number of lines for better display.
  
  IMO, unified memory makes programming a lot easier.
  
  Notes:
  1. In linux, what's the easiest way to find the smallest prime larger than a given number?
  2. To find the number of blocks needed for N threads, you can do it the Stanford way:
    
    grid_size = num_elements / block_size; if(num_elements % block_size) ++grid_size;
    
    or you can do it the RPI (i.e., my) way:
    
    grid_size = (num_elements + block_size - 1) / block_size;

5 Managed Variables

Last time we saw 2 ways to create managed variables. They can be accessed by either the host or the device and are paged automatically. This makes programming much easier.

Create static variables with __device__ __managed__. See /parallel-class/stanford/tutorials/vector_addition2.cu on parallel.
Use cudaMallocManaged. See /parallel-class/stanford/tutorials/vector_addition3.cu on parallel.
In either case, you need to call cudaDeviceSynchronize(); on the host after starting a parallel kernel before reading the data on the host. The reason is that the kernel is started asynchonously and control returns while it is still executing.
When the linux kernel gets HMM (heterogeneous memory management), all data on the heap will automatically be managed.
The reason is that virtual addresses are long enough to contain a tag saying what device they are on. The VM page mapper will read and write pages to various devices, not just swap files.
Any CUDA example using cudaMemcpy is now obsolete (on Pascal GPUs).

6 Doc

Nvidia's CUDA programming guide is excellent, albeit obsolescent in places. The Pascal info looks like it's been tacked onto an older document.

The whitepaper NVIDIA GeForce GTX 1080 describes, from a gaming point of view, the P104 GPU, which is in the GTX 1080, the card in parallel.ecse.

NVIDIA now has a higher level GPU, the P100, described in the P100 whitepaper and P100 technical overview. Note that the P100 is a Tesla (scientific computing) not a GeForce (gaming). This description is much more technical.

7 Managed memory issues

I'm sometimes seeing a 2.5x speed reduction using managed memory on the host, compared to using unmanaged memory. Dunno what's happening.

8 Misc hints

8.1 Vim

To get vim to show line numbers, create a file ~/.exrc containing this line:

:se nu

It will be read everytime vim starts, and will set line number mode.

PAR Class 5, Wed 2018-02-14

W Randolph Franklin, RPI — Wed, 14 Feb 2018 05:00:00 GMT

Table of contents

1 Optional Homework - bring your answers and discuss next week

Write a program to multiply two 100x100 matrices. Do it the conventional way, not using anything fancy like Schonhage-Strassen. Now, see how much improvement you can get with OpenMP. Measure only the elapsed time for the multiplication, not for the matrix initialization.

Report these execution times.
1. W/o openmp enabled (Don't use -fopenmp. Comment out the pragmas.)
2. With openmp, using only 1 thread.
3. Using 2, 4, 8, 16, 32, 64 threads.
Write programs to test the effect of the reduction pragma:
1. Create an array of 1,000,000,000 floats and fill it with pseudorandom numbers from 0 to 1.
2. Do the following tests with 1, 2, 4, 8, 16, and 32 threads.
Programs to write and test:
1. Sum it with a simple for loop. This will give a wrong answer with more than 1 thread, but is fast.
2. Sum it with the subtotal variable protected with a atomic pragma.
3. Sum it with the subtotal variable protected with a critical pragma.
4. Sum it with a reduction loop.
Devise a test program to estimate the time to execute a task pragma. You might start with use taskfib.cc.
Sometime parallelizing a program can increase its elapsed time. Try to create such an example, with 2 threads being slower than 1.

2 Nvidia conceptual hierarchy

As always, this is as I understand it, and could be wrong. Nvidia uses their own terminology inconsistently. They may use one name for two things (E.g., Tesla and GPU), and may use two names for one thing (e.g., module and accelerator). As time progresses, they change their terminology.

At the bottom is the hardware micro-architecture. This is an API that defines things like the available operations. The last several Nvidia micro-architecture generations are, in order, Tesla (which introduced unified shaders), Fermi, Kepler, Maxwell (introduced in 2014), Pascal (2016), and Volta (2018).
Each micro-architecture is implemented in several different microprocessors. E.g., the Kepler micro-architecture is embodied in the GK107, GK110, etc. Pascal is GP104 etc. The second letter describes the micro-architecture. Different microprocessors with the same micro-architecture may have different amounts of various resources, like the number of processors and clock rate.
To be used, microprocessors are embedded in graphics cards, aka modules or accelerators, which are grouped into series such as GeForce, Quadro, etc. Confusingly, there is a Tesla computing module that may use any of the Tesla, Fermi, or Kepler micro-architectures. Two different modules using the same microprocessor may have different amounts of memory and other resources. These are the components that you buy and insert into a computer. A typical name is GeForce GTX1080.
There are many slightly different accelerators with the same architecture, but different clock speeds and memory, e.g. 1080, 1070, 1060, ...
The same accelerator may be manufactured by different vendors, as well as by Nvidia. These different versions may have slightly different parameters. Nvidia's reference version may be relatively low performance.
The term GPU sometimes refers to the microprocessor and sometimes to the module.
There are four families of modules: GeForce for gamers, Quadro for professionals, Tesla for computation, and Tegra for mobility.
Nvidia uses the term Tesla in two unrelated ways. It is an obsolete architecture generation and a module family.
Geoxeon has a (Maxwell) GeForce GTX Titan and a (Kepler) Tesla K20xm. Parallel has a (Pascal) GeForce GTX 1080. We also have an unused (Kepler) Quadro K5000.
Since the highest-end (Tesla) modules don't have video out, they are also called something like compute modules.

3 GPU range of speeds

Here is an example of the wide range of Nvidia GPU speeds; all times are +-20%.

The GTX 1080 has 2560 CUDA cores @ 1.73GHz and 8GB of memory. matrixMulCUBLAS runs at 3136 GFlops. However the reported time (0.063 msec) is so small that it may be inaccurate. The quoted speed of the 1080 is about triple that. I'm impressed that the measured performance is so close.

The Quadro K2100M in my Lenovo W540 laptop has 576 CUDA cores @ 0.67 GHz and 2GB of memory. matrixMulCUBLAS runs at 320 GFlops. The time on the GPU was about .7 msec, and on the CPU 600 msec.

It's nice that the performance almost scaled with the number of cores and clock speed.

4 CUDA

4.1 Versions

CUDA has a capability version, whose major number corresponds to the micro-architecture generation. Kepler is 3.x. The K20xm is 3.5. The GTX 1080 is 6.1. Here is a table of the properties of different compute capabilities. However, that table is not completely consistent with what deviceQuery shows, e.g., the shared memory size.
nvcc, the CUDA compiler, can be told which capabilities (aka architectures) to compile for. They can be given as a real architecture, e.g., sm_61, or a virtual architecture. e.g., compute_61.

Just use the option -arch=compute_61.
The CUDA driver and runtime also have a software version, defining things like available C++ functions. The latest is 9.1. This is unrelated to the capability version.

4.2 Stanford lectures

On the web server.
On geoxeon: /parallel-class/stanford/lectures/
Lecture 4: how to
1. cache data into shared memory for speed, and
2. use hierarchical sync.

4.3 Misc

With CUDA, the dominant problem in program optimization is optimizing the data flow. Getting the data quickly to the cores is harder than processing it. It helps big to have regular arrays, where each core reads or writes a successive entry.

This is analogous to the hardware fact that wires are bigger (hence, more expensive) than gates.
That is the opposite optimization to OpenMP, where having different threads writing to adjacent addresses will cause the false sharing problem.
Nvidia CUDA FAQ
1. has links to other Nvidia docs.
2. is a little old. Kepler and Fermi are 2 and 3 generations old.