Comments on: Computing Power Skirmish

By: ET

ET — Sat, 09 Jan 2010 04:58:44 +0000

This reminds me a way to do it in perl. It is as simple as creating 8 sessions in the beginning and use each of them sequentially in the loop for data processing. I’m almost sure this works…

No need to do it now though, as the program has triumphantly ended.

By: Wei

Wei — Wed, 06 Jan 2010 21:50:06 +0000

No, I meant you implement the multi-threading part in your own code. This is pretty easy to do with C# or Java.

Anyway, if you seldomly need to do that much computation, maybe it is not worthy to make your program more complex. Also, 10 hour is not that long because it can overlap with your sleep, lunch, lectures, etc.

I once thought about writing a simple distributed computing framework for Stata to utilize all lab computers during the night to speed up my computation. I didn’t do that because I haven’t encountered heavy computing tasks after my labor econ course project.

By: ET

ET — Wed, 06 Jan 2010 16:15:38 +0000

so you are suggesting adding that “thread_concurrency=8″ option in my.ini? From what I read, it only works in Solaris machines. Anything involving rebooting MySQL is out of the question now, I’m actually quite happy with the speed, it should be finished in less than 10 hours.

By: Wei

Wei — Wed, 06 Jan 2010 15:56:07 +0000

Interesting.

I used to do some analysis over 30 years of labor econ data. I ran a regression for each year of data. If your job is similar to this, maybe you can very simply bring in some concurrency to “fully utilize” the computing power:

(1) Write the script that takes input of the year, and do analysis for a single year.
(2) Use some simple multi-threading to make the active threads to be fixed at X at any time. X = 8 for your dell server.

I guess that will make the computation faster.