Speculative Solutions

are good. Unfortunately they are not the end point.

There are two primary reasons I come up with a speculative solution to a problem:

1) I haven't encountered it before
2) I think we could be doing better than our current solution

I often need to remind myself that the speculative solution is not the answer, it it the starting point. It is a chance for me to learn and figure out if this is right. If it is the wrong answer I have two options:

1) Modify what I'm doing
2) Try something else

To do this I need to be in a position where I can change. If beginning my solution requires me to go to the end before I can evaluate whether or not it worked, I'm in big trouble.

Holidays

Looking forward to some holidays. First couple of weeks in July I'll be in Ottawa. Some time at the inlaws farm, rent a cottage with my family and of course a little talk on Agile development at CGI Ottawa for my brothers group.

John Duimovich

I worked for John back when computers were made of stone with little hampsters used to power them. He was one of the key people at OTI and really, really smart. I was one of the junior developers and really, really dumb.

John is one of the brains behind the IBMs (e.g. OTI) Smalltalk and Java VMs. He groks computers like no one I've ever met. His insight into how people program and how programs work is amazing. He's fun to be around as his mind works in weird and wonderful ways that make you "Wish I had said that".

Take for example his blog on "Visible ink"

http://duimovich.blogspot.com/

Nice to find his blog.

Blogger is pissing me off...

Blogger has identified my blog as SPAM. Which probably isn't far from the truth, however...

It's really annoying that I have to enter some word verification everytime I want to post. I've asked them to turn it off, it's been more than a week (took me a while to figure out what had happened). It's annoying.

Vlad: On risk and performance

Vlad (or Vald:) :
"Say there's a 99% chance that performance problems can be dealt with effectively late in a project, but there is 1% chance that performance problems can't be dealt with effectively. Is it worth risking an entire project on faith that one will find ways to fix the problems?"

Yes, on some projects performance is a severe risk. For example, my brother in Palo Alto needs to grab data from the partical accelerator as fast as possible and store it in a database (vast oversimplification). He worries about performance every second he programs. He writes custom ethernet drivers, uses customized hardware, reprograms switches and even writes in line assembly language in his C code. They spent 3 years making sure the software would be fast enough. Their risk was being able to do it fast enough, not whether or not they could store data in a database.

What is the risk on our project? Are we going to have to rewrite switches and introduce specialized hardware? Or just write clever java code? Our team is full of people who can write clever java code, it's not really a risk. I don't think it's even 999 to 1. Especially, when our code is trying make the business rules clear, and our tests form a living specification of the application. Acutally, I don't see a scenario where the talent on our team couldn't make our app performant, it's what we do and do really well. It is a technical problem, and we're really really really good at technical problems.

Our risk is the business and understanding how to make it work. Codifying the rules of production accounting is really really hard. Getting the specification right (the tests) and making it work correctly (the implementation) is a huge risk. How do we take a complicated body of (and not necessarily consistent) rules and make them into a reliable program? What are we missing that will make it hard to run properties? How does our model fail to meet the needs of our production accountants? We mitigate these risks by having production accountants test the application everyday. We write tests to create a living specification of what we've learned. We focus on these things, because if we get them wrong, we're in big trouble.

Incidentally, we have done back of the envelope calcs and shooting tracer bullets through the application to see where we need to be and actually are in terms of performance. In our heavy processing areas, our early calcs show we need to get 2-10x faster, depending on the area. Assuming, that is, we don't turn this into a batch process system, which would make us about 3x slower in the worst case. We can deal with that.

Vlad points out my oversimplification

"Does everyone really say that performance tuning should be saved for last? "

Well ok, I meant deferred as long as possible, not dead last. People advocating this strategy include: Kent Beck, Martin Fowler, and I believe Ward Cunningham.

Martin Fowlers discussion
http://www.artima.com/intv/tunable.html

Kent Becks discussion
http://www.rolemodelsoftware.com/moreAboutUs/publications/articles/lazyopt.php

My experience:

On my last application we spent the majority of our time implementing the system and deferring performance. It took three developers six months to build. It was 10x too slow. It took 1 developer 3 weeks to rip out the guts of well factored code and put it back together fast enough.

Near the dawn of time, when I worked at OTI, there was a huge panic after IBM built Visual Age on top of OTI/Smalltalk because it was *WAY* too slow for them. One (Senior) developer, Steve, with the right tools(profiler and an open mind) went and assessed the situation. In a few hours things were all tickety boo. It was simply the way they were initializing their objects.

In my experience, with the code teams I've worked with, deferring performance saved time and money. It's the old 80/20 rule (80% of the time is spent in 20% of the code). In my experience, I think this is way closer to a 95/5 rule.

If you change (refactor) your code alot, that 5% will change alot too (it will be the heart of your system). How easy would allocation be to change if it had to meet performance criteria as well? How could we experiment?

Garbage Collection

It's been close to 15 years since I worked with the garbage collector in OTIs Smalltalk. I was tasked with experimenting with parallelization for SMP boxes in OTI/Smalltalk. At the time, people were struggling with parallelizing garbage collections. The train algorithm was an interesting paper, suggesting better ways of parallelization.

I was simply curious to see if from a 50,000 foot view garbage collection has changed much in the last couple of decades. Apparently not.

This was a really good overview for me.

http://www-128.ibm.com/developerworks/java/library/j-jtp11253/

Empirical Performance

Everyone tells you to do performance tuning last. Why?

Because you can make performance tuning empirical. If you've done your tests first you'll be able to change the code to be performant, without breaking it.

The following process works for me as follows:


0) Prioritize and Instrument

Record how long things take in your app. This can be as simple as paper and pencil, or as sophisticated as automated performance tests suites.

Look at what is slow. Where would you get your biggest bang for the buck? What would make your users the happiest? Where will the load be in your system (back of envelope math works well here)?

Are they having trouble with the GUI, or the calculations taking to long? Which page? Help them.

1) Investigation and Hypothesis

You need to understand what the application is doing and how it does it. Use a profiler such as JProfiler. Eclipse has a free but slow one (tptp).

Look at time spent in methods - can you reduce the time?
Look at who is allocating objects - can you reduce the number of objects?
Look at number of calls to a method - does it make sense?

Hypothesize where you can save time, have rough guess of proportion of time it will save.

Example: Alan and I found a method was called 20x more often than we expected. It was a bug in the code that was doing unnecessary calculations.

2) Experiment

Do actual timings on the code you are running (I use println like utilities). Youcan't use a profiler for this because of dialation effects (e.g. the OS and Database are not being slowed down, only your java code - the profiler may be lying to you).

Do back of envelope calculations to see if what you're doing makes sense.

Calculate: Theoretical maximum you can save, and likely amount you can save.

For example: assume you find a method you can double in speed. Wow that seems like alot of savings. So lets say this method is taking 10 seconds out of a 100 second process.

Now lets assume you speed it up so it takes zero time:

Theoretical maximum: 10/100 = 10%. Will that satisfy your users?
Likely savings (Double speed of method) = 5/100 = 5%. Will your users notice?

3) Conclusion

Is it worth doing?

If you know your users need a lot more speed 5% won't cut it, you'll have to go back to investigating and finding where to improve.

Otherwise implement the change and then measure your actual speed gain.


Empirical Performance improvements. I find them worth the wait.

Useful little tool

For those of you who are part-time SQLers and need to format stuff occasionally.

http://www.wangz.net/cgi-bin/pp/gsqlparser/sqlpp/sqlformat.tpl

Service what you build

I always try to help service what I build.

If you were to buy a car from toyota - you expect to be able to service it there. Same with a toaster.

This analogy only goes so far, because when someone builds a toaster, they usually build 1,000's (or millions). Servicing can be done by specially trained servicing staff.

When I develop software, my customers will need service. I provide that service.

Specification Based Factory

Cool stuff today.

Simon and James came up with a new use for the Specification pattern. They combined it with the Factory pattern.

Basically, we need to find a class under some pretty heavy conditions (OK maybe we need a strategy pattern). They look up the class in a Map, indexed by a Specification object. They iterate over the map evaluating the specification, when one return trues they poof up a new instance of the class contained in the value.

Nice explicit code, with each rule contained in its own specification object.

I love simple stuff.

Process and people

Had an interesting discussion with Rado the other day. We were discussing process.

Just reinforced the notion that process is there for the people, not the people for the process. As such the process should accommodate all people and their working styles, not the other way around. When running a project I must try and make the process we have chosen work for all the people on the team. The inability of a process to adapt to an individuals working style is a failure of the process.

In general I try to adapt the process to the people. In reality, at times I am not able to adapt the process to meet the needs of all the people. This is a failing of the process and myself.

Performance techniques

Techniques we are using to improve performance:

1) Do more at once in a transaction.

With coarser grained transactions, we spend less time looking up stuff in caches and merging changes and more time actually doing the important work.

For example: Our importer was importing 1 measurement at a time. By grouping the measurements together on a per facility basis and updating them in a single transaction we reduced the time by 50%. DEspite caching

2) Cache.

Yep. It works. In our case we cache our snapshots (www.martinfowler.com/ap2/snapshot.html) which are constructed frequently in our networks of facilities.

For Example: Had one process come down from an hour to process to mere minutes.

3) Convert dynamically calculated results to stored results.

For example: Our statuses were being calculated all the time for complex large networks. Made the GUI slow. Store the results and poof things sped up by a factor of 5.

4) Convert stored information to dynamically calculated.

For example: We currently store some summary numbers for one of our facilities. Everytime something changes we redo our calculations. They end up being done over and over.

Instead, we make it an on demand calculation and our performance goes way up.

Many times we are pushing the time around, from a place where it hurts to one where it is OK.

Performance tuning in an agile world...

I have taken over the lead on the performance tuning of our application.

A few interesting notes on performance tuning:

1) We do performance tuning test first.

We use Watir to drive our performance tests - because they reflect the experience of the user. We can measure the responsiveness of the app. Also we get to show the users what we are measureing and see if it matches their workflow in a month

On our project, we have moved away from the traditional pass/fail test for performance. We have red, yellow and green. Green means we're with in our target, yellow indicates we're within about 50% and red, well, we're in trouble. It helps prioritize our work.

As always we're running these tests a few time a day in a cruise-control like process.

2) There are different areas of performance tuning.

We have GUI response time, processing time and import/export time (our application has a large component of reporting and third party application interaction in it).

We concentrate on each one, writing tests and getting feedback from our testers on how the system is doing.

3) Break down the problem. Tackle one thing at a time to make it managable.

Performance, Load, Concurrency are all heavily interrealted problems. The nice thing is that we can takle them individually, we don't have to deal with them all at once.

We use both real and imaginary data to drive our performance tests (Actually, we use real data that are part of our regression suite).

4) Thank god that your developers have been refactoring code. You'll notice when you have some "original" design code, versus stuff that has been worked over. It is way easier to tune the latter.