Performance tuning in ernest

Up until this point we have been performance tuning for testing. Our users have been testing the application all along, and our goal has been to keep them happy with the performance of the system (not ecstatic).

Now we are nearing the end of our mamoth, huge, horrifically large project (why did I agree to help run this?) and things are starting to stabilize. We're five months away and seriously testing our system. Data structures and core internal pieces are pretty clear and we can optimize the living crap out of them. And optimize we shall.

Production accounting has a few unique attributes to it:

First: If you give us 7 numbers, we can very quickly turn those into 7,000. For example, processing 20 wells results in 72,000 numbers (or allocations) being created. Our biggest customer has 40,000 wells. (144,000,000 allocations per month in the work case - note this is the upper bound). Huh.

Second: Accountants have four (important) deadlines a month. For one of those deadlines (Oil/NGL Splits) we need to generate all of the data. Best of all the data for all their systems will probably only be available the day before the deadline. Don't ask, the production accounting business is wacky. We will have to be able to do all our allocations in about 12 hours.

We need to write 3400 numbers (allocations) a second.

We're out by about a factor of 10 (we do around 300-400/sec on a single Intel CPU box). That is not very far, especially as we are just beginning optimization. Better yet, we can expect between 8 and 32 cores doing the work on the boxes.

So what did I miss on my envelope?

We'll have fun, fun, fun 'til daddy takes our profiler away....