How does threading save time?-Collection of common programming errors

Most of the comments you have are correct, but I’ll throw my two cents out as well (and list the comments in here):

Jonesy: “threading is most efficient in multi core environments” -> Yes, but this is a single core cpu…so I’ll come back to this.

KooKiz and John Sibly: They both mention I/O. Your machine is not crunching along at full power 100% of the time. There are quite a lot of other things happening that take up time, and during those events, your CPU gets to take a break.

(point of reference: I/O can be network transmission, hard disk/RAM read, SQL query, etc. Anything that brings in new data to the CPU or offloads data FROM the CPU)

These breaks are time that your cpu can be doing other things. If you have a single core cpu (we’ll ignore hyperthreading at the moment), and a single-thread application, then it runs happy as can be. It does not, however, run constantly. CPU scheduling will give it a cycle or two, then move on to something else, then after a while come back to your program, give it a few more cycles, move on, etc. This gives the ILLUSION of being able to do “multiple things at once” on a single core cpu.

Now, since this is a normal program and not some insanely small assembly program where you are writing values to the cache directly, your program stores data in RAM…a relatively slow storage medium compared to the CPU cache. Because of this, loading values takes time.

During this time, it is possible that your CPU has nothing better to do. This is where you can see a speedup on a multi-threaded application, even on a single core. The other thread will fill in those extra CPU cycles, where the CPU would otherwise be idle.

Note that it is highly unlikely that you will see a 2:1 speedup. It is much more likely that your 2-threaded program will only see a 10-20% speed boost, if that. Remember, the “other” thread (which at any given point is the thread that is NOT performing I/O) will only really be running at its full capacity while the first thread is performing I/O.

Frequently, however, you can actually see a WORSE time. This is because your CPU now has to spend more time switching between the threads in your process (remember, we can only run one thing at a time!). This is called overhead. The second thread creates more overhead than it can make up for, and so the process overall slows down.

On a multicore machine, you have two physical executors…which means the second thread gets an entire new core to work with. This means that it doesn’t have to compete with as many other things for execution time. Therefore, we get a substantial speedup here.

Then of course you have multiprocess programs that execute over a cluster, but we’ll save that for another time.