Multithreading is used for two main reasons; one is to increase throughput and/or decrease latency, and also increase scalability and other is to make the code more succinct, structured, and the separation of concerns.
Using multithreading to increase throughput and decrease latency is very tricky. It is often counterproductive because it is not implemented right. Unfortunately, it is very hard to have a generic multithreading solution for all problems. It is heavily dependent on the nature of the problem and the hardware/software platform. That is why the multithreading solution in DataFrame is very tunable and requires careful user adjustments. I suggest to always start with a single thread and later when the system is working correctly experiment with multithreading. DataFrame gives you the tools and plumbing to do that.
Also, watch this video: Scott Meyers: Cpu Caches and Why You Care