Year in review

I've had a lot of fun this year. First and foremost, I enjoyed my family, my daughter (Rosie) is beginning to read and my son (Charlie) has found the joy of hotwheels. They're both sponges learning about everything around them, it's all new to them. It's good to see them enjoy life so much and so intensely. Amy has begun volunteering at their school helping the kids in our city. Again far more important work than developing applications.

I'm really enjoying my job, and owe a big thanks to Darrell who refered me to the job. In the beginning, there was just Colin, Joseph and I. For about three months we had the ability to architect and design the system. So we did the minimum we could get away with for architecture and design and started building it. A lot of fun. I've learned alot more about XP on this project, and code is starting to become like putty - something you can mold and reshape.

This has turned into one of the larger projects I've lead - we're heading to 24 developers soon (Holy Crap). Very few cracks are appearing in the code although it isn't perfect. Good challenge to figure out how to keep the team humming. I think Colin, Joseph and I have our work cut out for us, we'll probably have to lead more (presenting and steering vision) and try to be less controlling as the team continues to grow. That won't stop us from coding and participating though :).

I had a really good year in terms of health, until the end. The new drug Enbrel transformed my life by removing a cane, lack of sleep, and the continuous pain I've experienced for the last ten years or so. I've had a few set backs in recent months, but the doctors are working with me to figure that out.

So thanks to Colin, Joseph, Cesaer, the PAS development teams (Mordor and the Shire), the BA team, the QA team and the customers, this project is fun!

Well, Happy New Year!

- ted

Testing Skunkworks

Built a FIT framework for Pas (our current project). Allow the user to enter a simple script in excel for importing and click buttons (using JWebUnit and direct calls to our facade). On another page the user enters the values they expect to see in views.

They run the test through JUnit, which creates an additional sheet highlighting in Red those values that are different than expected.

The last time we tried FIT the developers were maintaining them in Java Code. The overhead was high and development is the bottleneck in our project. I'm hoping these new FIT tests can be maintained economically by the testers. We'll see.

Tests deter change...

When you write tests they cause code to become less maleable. They define how the code must behave.

Functional test (or user facing tests) tend to make requirements harder to change because to change a requirement you must change a test.

If your functional test reflects your implementation - it can be harder to make changes in the design of your application, because you must change the code that reflects your implementation. So, make sure your test reflects the domain rather than the implementation. (createAnonymous' and custom asserts are essential here).

Unit tests have the same effect. They are useful for thinking about how you will design a feature, but get in the way once you decide to change the design.

Use them carefully and wisely.

Test only code

Just found some test only code in our system. Freaks me out. We have a test facade that allows people to access data that is not normally visible through the facade. For example, some results are not visible on the GUI, they are only reported in the database, or in reports.

When you need access to that data for testing purposes, you can write a test facade method. In general though it should not be used for any business logic. It can appear in setup code and verify code, but it should never appear in the execute (system under test) area of a test.

Looks like I'll be doing a refresher on Jan 3rd.

Brian Merricks when to automate paper

http://www.testing.com/writings/automate.pdf

.. courtesy of Joseph King.

Directed at QA people. A fairly long paper that says you should only automate those tests that are going to help you.

Interestingly, it ignores the role of testing in development - especially in an agile environment where things are changing constantly. I think the tradeoffs change quite a bit in this environment. Especially as tests give you confidence that things are OK. Confidence gives you permission to make both small and large changes to the code base.

QA Testing Faces two ways

When QA is testing an application in an XP environment they serve at least two masters. For development they verify existing functionality to see that it conforms to the requirements and integrates with the rest of the application. For customers QA examines the application for missing requirements.

A couple of tools in the hands of a good tester are:
1) Exploratory testing http://www.testingcraft.com/exploratory.html
and
2) Regression testing http://fit.c2.com/

I'm trying to adapt a fit style testing harness to our project. In essence I realized I'm creating a mini macro language for PAS. It does imports, a small number of functions through the facade and UI (ala jWebUnit) and then a small number of asserts.

Key to success is keeping it simple/small. It might be maintainable. We'll see.

The Trinity

Refactoring, Patterns and Automated Testing have always been important, but separate thoughts in the community.

As I continue on this project they are becoming more and more tightly coupled.

Testing leads to code. As the code becomes more mature, I begin to see patterns that apply (or often go looking). When the pattern is clear, I refactor. As I refactor (using the tests to save my ass), the patterns continue to emerge. Often this leads to me thinking about more tests that need to be written, because now I understand how the business works in the code, and deviant cases become more obvious.

I stop before the code is right. At some point the ROI on refactoring is no longer worth the effort. Besides it will change next sprint anyways.

ROI

ROI isn't just about money, it also applies to refactoring and redesign in your code.When you refactor you need to think about what the cost is and what you get. Don't forget the risks and costs.

Risk

• Will the refactoring help
• How long will it take
• Will it introduce more bugs

Costs

• Time away from building new functionality
• Code changes others must figure out
• Time/effort to learn the refactorings necessary to implement the change

Benefits

• Future features can be implemented faster
• Patterns in your system may become clearer for others
• Bugs may be more obvious in the refactored code
• The code may become maintainable (readable)

Ask yourself: Do the benefits outweigh the costs?

In my current refactoring I'm clarifying the Snapshot pattern ala Martin Fowler in our code. We have a mixture of Snapshot patterns (delegates and facades) in the code and the business logic is spread between snapshots and their backing objects. Our ROI for this refactoring has been as follows:

Benefits

• Bugs/duplicate code around date effective logic. No separation between valid data and valid data for a date. This should clean that up.
• No clear patterns for implementing the next date effective thing, so we'll get another new one, with a different set of bugs
• More maintainable. Some of the common code for date effectiveness is being pushed up to super classes.

Costs:

• We're costing (in terms of time) more than I'd like to do this, so we'll reevaluate tommorrow
• We had to take on some extra refactorings to make this possible (ExtractSubclass to be able to move the business logic)

I believe its worth it in this case.

Nice refactoring site

http://refactoring.be/

Different ways of looking at refactoring.

Refactoring from A CIOs perspective...

Nice non development centric view of refactoring. Good read.

http://cio.idg.com.au/index.php?id=1688864994

Hi Gus!

Apparently blogs are good for something. A life long friend, whom I'd lost touch with after having kids sent me a Christmas card this year, after he found my blog.

I'd gone googling him but had only found a reference to a "meeting" he had attended. Now I have an address, you'll hear from me soon. If I can figure out Gus' email, I might be able to send some pics.

The simplest things...

... make me happy.

Rado came up with a simple fixture for setting up tests on our project. After he created it, it was stunningly simple and will allow us to remove a whole pile of code from each of our tests. Things this simple reflect great insight and ability. Thanks Rado.

Cool. Team is picking up its own steam.

Agile Architect

What's an agile architect? I used to think there was no place for an architect on an agile team. Working with a large team my opinion has changed slightly. They are not an architect, sitting off in their own room coming up with "architectural" memos.

On an agile team the architect is one who takes time to review the code base as a whole and ensure it is consistent, following patterns and of sufficient quality. They will spend time training individuals on the team and helping to guide the code. They worry about making the past better rather than the future possible.

Things I would look for in an Agile architect are:

1) Experience with standard patterns (it takes a while to figure out how to properly apply patterns in a project)
2) Experience with large refactoring (this is a skill that takes years to get good at)
3) Experience with testing and maintaining tests
4) Someone who strives to keep things simple.
5) Someone who can mentor
6) Experience on a project similar to the size you're working on.

Technical Scaffolding

Well for the moment I'm happing with the idea that Technical facing tests are Scaffolding. They can help you think about the solution you're coming up with, but they are essentially stuff you don't need to keep. I certainly use them that way sometimes - test a small piece of complicated code.

Here's another persons take:
http://www.refactoring.be/articles/tdd-top-down-vs-bottom-up.html

Mark needs a Pronto

Mark bought a home theater. It has a gazillion buttons with a billion remotes no doubt. I have the same problem.

My problems were solved with a Pronto remote. It's extremely geeky in that you can (have to) program it for hours to make it work well. However, you can create a small simple minded UI on it, which most of the other remotes fail to do. It is also easier to program macros and stuff like that. Best of all it has a huge community (http://www.remotecentral.com) around it who have done most of the work and you just need to customize their existing screens.

If you're a real geek you can download the application and run the simulator before you buy.

PVR

Bought a DCT6208. Changes the way you watch television.

Even my wife has figured out the remote in order to get some of the features. Way too convenient for anyone who can wait twenty minutes before watching their favorite programs. On the other hand HDTV is great quality but probably not different enough for people to switch their equipment.

Guaranteed wrong Prediction: PVRs will catch on like DVD players.

Change builds on Strengths

I had an interesting insight today while talking to a friend: Change is good, but it must incorporate current strengths.

On the XP bandwagon, it is easy to tell people "how to do it right". There are certain accepted practices that help make your project a success and you need to adopt them. Getting to the point where people are willing to adopt them and do it right is an interesting challenge all on its own.

Many projects do many things right, build on the things they do right. Introduce things at a pace the people can handle. Sometimes massive change will be possible, sometimes incremental change over a long period of time. This was the mistake I made at presslogic. I wanted them to make change at my pace, not theirs.

Success should be measured as improvement, not conformance.

Unit tests and large refactorings

I was doing a large refactoring today, clearing up the Snapshot pattern (ala Martin Fowler) in our system today. The refactoring is separating responsibility and changing some messy code into some cleaner code.

About half the time I was fixing the code. The other half of the time was fixing the tests. Some were badly written tests, which the cleanup revealed. Alotof the time was dealing with Unit tests, which made some assumptions about the implementation (because, well, that is what they do).

The unit tests were of no use in the refactoring - the business facing tests however revealed all.

People at work are calling me the Unit test bigot.

Fn F2

If you hold down the "Fn" key and F2 on a dell it turns off your wireless card...in the bios. There is no indication that this has occurred other than you can't see the network. Windows XP still shows the card as completely functional otherwise. I spent a couple of hours and finally had to spend an hour on the phone with support to figure out why my wireless card had stopped.

Stupid design.

Test Update

2700 Tests running.

Tried unit testing (Brian Merricks Technical Facing tests) again. I still get no satisfaction from them. They are way to geeky. They tell me how it does it, seems redundant to self describing code.

I haven't been able to have them make me code faster. Just get lost in details. Ah well, I'll keep reading about them, maybe I'll meet someone that changes my perspective on them.

Here's what I've been told about unit tests that haven't worked for me:

Defect Triangulation: Unit tests are supposed to help you localize exactly where the problem is in the code. Yeah, but usually the code hasn't been wrong (and hence the unit tests still pass). The problem is I missed something conceptually - so my unit test passes my functional test fails and I'm still in the debugger letting the numbers tell me what I missed conceptually.

Design Quality: By writing unit tests you will get better designed code, because you must break it down into smaller testable pieces. My designs evolve over time - usually when I'm writing code the first time the design sucks because I don't really understand. The unit tests just make that sucky design more of a pain in the ass to change.

Are unit tests only for smarter people who can get it right the first time? What am I missing?

Three biggest changes in my profession in the last 10 years...

Things that change our field are not generally the mechanics (e.g. languages), but how you do things.

Lego. We only have fours and eights but at least we don't have to derive everything from first principles. Patterns have made things a little bit easier.

The Broom. We can now clean with something more than a pair of tweezers. Eclipse and IntelliJ have given us refactoring tools. I expect they will be part of all development environments in the next five years.

Check it. We now check to see if what we did worked. Automated tests have taken the profession by storm.

Guess I'm on a large project...

Well Mark (http://www.markroseman.com/blog) has pointed out I'm on a large project. Hadn't really thought of it that way. Well I knew it, but it wasn't really in the plan to help run a "large" project.

Basically, I was referred by Darrell Hull, whom I really enjoy working with, got to meet with a smart and clever person (Colin Jones) who needed to build an application for Oil and Gas. Colin was fun to talk to, seemed to have the same attitude towards work that I did, and was interested in exploring the whole Agile thing with me.

My career planning can pretty much be summarized as if you have a cool project, you're nice and you're willing to trust your developers I'm with you.

Awesome...

The smaller team has been growing and maturing.

Zach and Trevor used template method to refactor our "balancing" code that has been around since the dawn of time. They figured out the template method refactoring and how to apply it.

The Mordor team (as christened by our brother team, the Shire) applied the "Compose Method" refactoring (from refactoring to patterns) all week. The point about the content of the method being at the same level is a good way to think about this. Many of our methods became smaller and more readable.

Your project is going live next month...

or your project is cancelled tommorrow...

Those are words we should all want to hear. It means that what you have today must be useful to your customer tommorrow. The stuff for the day after tommorrow will take care of itself.

If you stick to that, you can avoid building stuff that will never be used. Some of the best developers I've known (e.g. Mark Roseman http://www.markroseman.com) have figured this out a long time ago.

Design and architecture become way less important. Working stuff becomes way more important. So does maintainable code. Feedback becomes way more important. The customer becomes more important. Which is an interesting addendum to Marks blog http://www.markroseman.com/blog/2004/12/engaging-with-customers.html.

So how do I teach this to my customers?

Profound Thoughts

I used to think I needed to explain things to the world, how it was. I started a blog that had long essays. I might do that occasionally now, but in general I just want to share my experiences.

A short description of an experience, positive or negative, is much more effective than a long diatribe. Hope this is short enough.

Shaking things up...

Just a reminder that shaking things up always has a consequence - usually stuff you didn't think of.

We split our team about 1.5 weeks ago. We have completed 57 Jelly Beans. Normally in a month the team completes about 100. Several reasons have been cited including: 1) a sense of competition, 2) different teams get stories that are appropriate to them.

Ordered a dell...

Bah, went to the local shops looking for a basic laptop. No one had one with a SXGA+ screen. Ended up ordering another one from Dell.

Just received an invoice that showed Dell credited me 100% of the broken laptop. Very nice. Guess I'll stick with them a little longer.

DTOs...

I read the analysis of the over use of DTOs on projects (http://martinfowler.com/bliki/LocalDTO.html). There are many good technical reasons not to use DTOs: cost of implementation, they are only be used when the cost of communication is high, etc...

All sound reasons.

I use them because they help me think. I'm not the smartest guy in the world and I make mistakes. Sometimes I put business logic in the GUI. Sometimes I put display logic in my business logic.

The simple three letters "DTO" at the end of my object name make me think and that, for me, is worth the time it takes to write the assemblers. If I'm in the business logic and DTO appears, I know I'm mixing layers. If I'm working with a DTO and there's a bunch of code around it - I might be trying to put business logic in. DTOs more than anything keep a careful watch over the layers in my systems.

I'm a simple guy, I make simple mistakes. DTOs point them out. Thinking is my bottleneck - not writing assemblers.

I manage because I have to

Interestingly different people have interpreted my changing from teamlead to developer in different ways. I have been asked sympathetically how it feels to report to somebody. I was actually dumbstruck by the question, I didn't see how others would interpret it as a demotion. Sometimes I'm thick.

I only manage because I want to work in a great environment. I view it as a necessary evil - if I manage then I can work the way I want to when I'm not managing. On this project I manage with people who manage things the same way I do.

With the team rearrangement (orchestrated by a group including me) there was only the need for one of us to manage. I drew the long stick and the other guy drew the short stick. He had to manage. I won. I get to do what I love without having to manage. Even better I know the great environment will be maintained by my new "boss".