Every AI revolution rides on the back of an HPC revolution

Today's critical challenge is how to do AI training and inference without destroying the planet with the energy requirement. Parallelism is an essential element of that. An energy optimal/efficient solution to any computing problem subsumes time efficiency (energy being the product of time and (average) power, a faster algorithm/program gives a multiplicative reduction in energy).

In this class you will learn how to make programs go as fast as possible on a number of platforms using a range of techniques. We will cover concepts of designing, writing, debugging, and analyzing parallel programs.

We will study the two main paradigms: shared memory (on CPUs and GPUs), and message passing on distributed memory machines. We will also examine a number of parallel algorithms for a range of problems and algorithm analysis using the work-span model. We will also examine a number of parallel algorithms for a range of problems.

This semester, CS475 will be combined with CS575, Parallel Processing. The lecture material will be largely shared, there may be special advanced lectures, covering material that is optional (extra credit) for students in CS475. In addition, each assignment will have such extra credit questions.

On completion of this course you will be able to:

• Write a number of programs in three C dialects supporting different parallel programing paradigms (OpenMP, CUDA and MPI).
• Compile the programs using standard desktop and clusters, and run them on CS department Unix platforms.
• Measure the running time as a function of problem size and number of processors.
• Analyze observations associated with the compiling and running of the programs.
• Report on your observations and analyses for strong and weak scaling.

In addition to these practical skills, you will also be able to

• Implement your skills in a on a substantial parallelization problem and/or investigate a research problem.
• Understand limitations to fully exploiting parallelism.
• Develop a deep understanding of parallel algorithms
• Realize the importance of (i) a better algorithm (asymptotic complexity), (ii) a better sequential program (i.e., implementation of the algorithm and (iii) better parallelization.

Hybrid undergrad-grad course

This is a hybrid course (called CSx75) combining CS475 with CS575. CS 575 covers the same material, but at a greater depth. The MW lecture material will be common. CS575 students will be tasked to do more with the material that is presented.

• Assignments will require additional effort/deliverables and deeper analysis.
• Quizzes and exams will have additional questions.
• Your project is expected to tackle a current research problem and be more inventive.
• You will have an additional lecture in the later part of the lab (Tuesdays 4:00 - 5:00 PM, in CSB 425).

CS475 students are welcome (encouraged) to do the additional CS575 work (for extra credit). Even late in the semester students will have the option to switch their registration.

CS 370 (System Architecture and Software), or CS 320 (Algorithms) or instructor consent or

Parallel Programming in C with MPI and OpenMP, Michael J. Quinn (Mc Graw Hill, 2003 ISBN 0-07-282256-2). This textbook is out of print in North America. If you are able to get it from any other source (eBay, Amazon, international, etc.) that is fine, but we have worked out an arrangement with the publisher to make it available to students. This will cost about $50, and can be purchased at the CSU bookstore.

Here are the formally graded elements of the course and associated weighting:

Activity	Weight
Programming Assignments	20 %
Tests (Midterm & Final)	40 %
Labs/Discussions/Participation	10 %
Quizzes/Homework assignments	10 %
Final Project: term paper	20 %

Semester grades are determined by the weighted sum of points earned in each of these areas. Tests will be done individually and grades assigned on an individual basis. The assignment of letter grades will be made as follows (we reserve the right to lower any of the cutoff points):

Grade	Score
A	≥ 90%
B	≥ 80%
C	≥ 70%
D	≥ 60%
F	< 60%

Midterm and Final: Make-up exams are only given for extraordinary circumstances (e.g., illness, family emergency). Students must consult with the instructor as soon as possible, before the scheduled start of the exam. Course examination dates are listed in the syllabus; be aware of them and plan accordingly.

Programming Assignments: Programming assignments will be submitted using the Checkin tab on this website. Specifics will be included in each assignment. Always check the assignment page for due dates. Late assignments will not be accepted; students not having submitted programs/reports receive an automatic zero on the assignment.

The project entails two options.

• Empirical study of parallelization of a larger application than the assignment problems.
• Identification and study of an open research problem.

Either option will also invlove writing a two-to-four page paper.

Midterm	TBD
Final Exam	Monday December 15, SB 110 (any two hour window starting 11:00AM)

Midterms and the final exam will be taken as Canvas quizzes and will be proctored.

Online students will need to arrange for a proctored location where they live. Please provide the information about this location to the instructor well before the exam date. The passcode that will release the exam will be given only to the proctor.

All students taking this course are expected to participate actively. For all students, this includes asking and responding to questions. There will also be canvas quizzes.

All students are expected to conduct themselves professionally. We (the instructors and GTAs) assume you are familiar with the policies in the student information sheet for the department. Additionally, you are computing professionals, albeit perhaps just starting. You should be familiar with the code of conduct for the primary professional society, ACM. You can read the ACM Code of Conduct HERE.

We work to maintain an environment supportive of learning in the classroom and laboratory. Towards that end, we require that you be courteous to and respectful of your fellow participants (i.e., classmates, instructors, GTAs and any tutors). In particular:

• Please turn off the ring on your cell phone. If you are expecting an emergency call, sit near the door and slide out discretely to take it.
• In class use of electronic devices in general, and laptops specifically, is permitted as a courtesy so that you may better participate and learn. If at any time the instructor judges that an electronic device is becoming a distraction the student may be asked to to turn it off and put it away.
• All exams and quizzes are to be done without the aid of any unauthorized aid. Phones, and all other electronic devices must be shut and put away during exams and quizzes.