Ted O'Grady on Agile Development

Serendipitous

2007-08-27T10:21:00.001-06:00

Barn Shadows

2007-08-27T10:07:00.002-06:00

Ghosts in the wood

2007-08-27T10:07:00.001-06:00

Art in Calgary Alley

2007-07-27T11:52:00.000-06:00

Reflected Wealth

2007-07-27T11:48:00.000-06:00

Taken from the south side of the tracks in Calgary

Relax

2007-07-10T23:39:00.001-06:00

Holidays

2007-07-10T23:36:00.000-06:00

Fixing one of the things that bugs me...

2007-05-30T21:58:00.001-06:00

One of the things that bugs me is how long it can take to set up the infrastructure for a small project. Purchase hardware, install OS, configure OS, CVS, bugzilla, webserver, cruise control, and, oh yeah, backups. Oh crap don't forget security.

Why do I need to do all this work everytime? True, it's kind of fun to configure this stuff and tweak it to your hearts content, but the installation and setup of it I could really care less about. On alot of small projects I have just seen people go without. "Do we really need cruise control?" "Do we have a security expert?" "Ralph will do it on top of his day job."

It is simple stuff that shouldn't be a burden on a project. It bugged me.

Now my partner and I have created ITForDevelopers. You can purchase a virtual server, preconfigured for development. Root access to a full system. Start developing the day your project starts rather than setting up infrastructure.

Where have I been?

2007-05-30T21:49:00.001-06:00

Doing too many things.

I'm PM'ing one project, leading a team on another, and then starting a new product. Learning, Learning and Learning.

Most importantly, I am heading toward a summer where I have two months off. Not something I've done since high school. I'll be spending it with my family and enjoying the sun and warmth which lasts far too briefly here in the Northern climate. We'll be returning to Ottawa to visit relatives and do some car camping around Alberta.

No big trip. Just my family. Nice.

Which Profiler?

2006-10-28T12:19:00.000-06:00

Jamie asks:

"Ted,What are you guys using for a Profiler? If you have any advice on a profiler
or, more importantly, common areas a profiler flags, I'd love to hear them."

Well we use:

JProfiler - for exploratory work (has the problem of dilation)
com.tedogrady.profiler - for precise timings (basically println)
The toplink profiler - for understanding the cache and gross database numbers
tkprof /explain plan- for understanding how queries are working in the database
math and common sense - for figuring out how to solve the problem

Performancing tuning is a highly empirical thing.

Establish a target time.
Make sure you have a test showing the slowness. Automate it.
Develop Causal Theory
Test Causal Theory
Develop Potential Solution
Experiment with Potential Solution
If 4 doesn't help, go back to 3
Implement Solution

JProfiler helps alot with the Causal theory. You can use it to explore your code as a whole looking for problem areas, testing out theories you may have developed about the code. But, if you are working in a database application, DO NOT BELIEVE EVERYTHING IT TELLS YOU.

It's lying to you. Remember it slows down your application, but not the database. So, hot spots and the relative timings it tells you are lies where concerned with the database. For example, the current piece I am profiling is spending half the time in the database dealing with foreign keys. The profiler told me my problem was in navigating some hierarchy.

I use Jprofiler for the following features:

Execution stack trace - shows the order in which things were called
Number of Invocations of a method - really useful (do quick math to see if the numbers make sense)
Percentage time in a method - also really useful
Hot spots - (rank methods by amount of CPU time/calls)
Memory profiler - which objects are sucking back the memory

I tried the eclipse one about 6 months ago (TpTp) but it choked on simple stuff. Haven't tried it since.

com.tedogrady.profiler implements println with extra special fancy do-dad stuff. It allows you to verify what your profiler is doing. It lies to you a lot less frequently than JProfiler. It's about 40 lines of source code that I got tired of writing on every project, so its now open source. It is in with my toplink in memory stuff, or just email me if you want a copy. Basically, it has a really small overhead to verify what you learned in JProfiler, or if Jprofiler is really lying to you then you can use it as a more painful exploration tool.

To profile a suspicious method you can do the following:

Profiler.start("Profiling myMethod()");
this.myMethod()
Profiler.stop("Profiling myMethod()");
Profiler.dump();

You can put as many starts and stops (each with unique tags) in your code as you want. It'll tell you nice things like the average time, cumulative time, invocation order, blah blah blah. Many of you are thinking map. You'd be right.

TopLink probably comes with a profiler because of all the people that complain that toplink is slow. Very smart of them to give you the tools to see where the slowness/idiocy is. The tool usually tells me again how stupid I really am. As an idiot, I frequently forget I'm an idiot. It's part of the condition.

The command that lets you show sql in toplink is also very useful. Dario has been using it to fine tune the SQL we're throwing at Oracle. It lets you configure all those wacky options in toplink and see what they do to the SQL (just what does batch write or batch read do? Gee, they make things a lot faster).

tkprof, explain plan, and oracle analyzer are useful when you have to think about the database. They tell you what the database is doing, like how long a query is taking, or the strategy that the optimizer is using to evluate your query. (Sadly, you do need to understand how oracle implements things if you want to tune it). asktom.oracle.com has all your answers pre-packaged.

I haven't gone looking for Oracle analyzer in Oracle 10g, but I've used it in 9i. Basically, it is a way for you to ask the Oracle optimizer what will work best. Rather than you analyzing (guessing) what the best index are for a query, oracle will recommend some. 90% of them are optimal, and some recommendations slow things down. Remember that whole test thingamy.

math and common sense are the best tools of all. This would be why you went to school. When you look at the number of invocations, and the time, does it make sense to you? What do the numbers mean? A scrap of paper, a pencil, and some simple multiplication and division can tell you a lot more about what your program is doing than any profiler.

Just as a side note, I learned most my techniques sitting across the hall from Steve Northover at OTI when he was writing the profiler for OTI/Smalltalk. Thanks Steve.

Performance tuning in ernest

2006-10-25T20:58:00.000-06:00

Up until this point we have been performance tuning for testing. Our users have been testing the application all along, and our goal has been to keep them happy with the performance of the system (not ecstatic).

Now we are nearing the end of our mamoth, huge, horrifically large project (why did I agree to help run this?) and things are starting to stabilize. We're five months away and seriously testing our system. Data structures and core internal pieces are pretty clear and we can optimize the living crap out of them. And optimize we shall.

Production accounting has a few unique attributes to it:

First: If you give us 7 numbers, we can very quickly turn those into 7,000. For example, processing 20 wells results in 72,000 numbers (or allocations) being created. Our biggest customer has 40,000 wells. (144,000,000 allocations per month in the work case - note this is the upper bound). Huh.

Second: Accountants have four (important) deadlines a month. For one of those deadlines (Oil/NGL Splits) we need to generate all of the data. Best of all the data for all their systems will probably only be available the day before the deadline. Don't ask, the production accounting business is wacky. We will have to be able to do all our allocations in about 12 hours.

We need to write 3400 numbers (allocations) a second.

We're out by about a factor of 10 (we do around 300-400/sec on a single Intel CPU box). That is not very far, especially as we are just beginning optimization. Better yet, we can expect between 8 and 32 cores doing the work on the boxes.

So what did I miss on my envelope?

We'll have fun, fun, fun 'til daddy takes our profiler away....

Something to try...

2006-09-17T14:08:00.000-06:00

When I read a book that I think is good, I like to try a few things out of it that you normally wouldn't (Why did you read the book if not to learn something new?)

I've been reading Peopleware by Demarco & Lister and really enjoying it (it lines up with a lot of the things I've been taught and try to practice). I'd like to try the "Power of Veto". Allowing our teams (development and customer) to state when their portion of the product is ready.

I think the question to ask is "When will x be ready for production?" where x is contracts, volumetrics or allocations.

Discovery phase timing

2006-09-06T10:46:00.000-06:00

"Ted,How soon before production do you think a team should go into a 'discovery'
phase?" - Jamie

As soon as possible, preferably at the start of the project. You want to make sure that your product is "production ready" from the day you start.

The later you wait to start the larger your backlog of bugs and missing features.

However, my reality has not always allowed discovery to commence with the project. Generally, customers do not want to explore a partially built system, they don't mind testing pieces of it, but have trouble seeing if all the pieces will hang/operate together. Often customers are shared between your project and another, their time constraints do not permit them to explore the application. Quality assurance people are really good at uncovering bugs, but it requires a real user to understand how the application hangs together in their environment.

There are many mitigation strategies I have used in the past:

1) Get it into production. If you have a truly green field application (no predecessor) you might be able to get it into production really early. The bare minimum of function can be useful to the end user. (Teamwave was good at this)

More often than not, there is an existing application, that does at least some of the same work that the new one will. In this case I try to duplicate the old application and put it into production once it has the same functionality as the old app. The second release is then the new enhancements.

Arguments I haved faced include: You will not be able to add new functionality as it will be just as brittle as the old app. It is too expensive for us to put it into production so we must add all the new features first. There are not enough features to be useful to a customer.

2) Get real data into the system early. As a substitute (or when getting into production is a long way away) this can work really well. Users get to see their data in the system and see what works and what doesn't. The data is meaningful, and having the users work with real life examples brings out all the complexity of real data. (Inform Alberta did this)

On a previous project we started migration a few months into the project so that data was readily available to the users to test with. They had lots of complaints, but also lots of learnings about how things worked. We got really good at migration, and did it multiple times over the remainder of the project as new functionality was added.

Arguments I have faced include: The data is too complex to migrate. We will oversimplify the problem, because the real data doesn't reveal all cases.

3) Eat your own dogfood. If you are really lucky, you can use the product as you produce it. Basically, you use your own product as you develop it. (Teamwave and OTI did this). OTI was actually a master of this, we built the Smalltalk language in Smalltalk, using our latest builds. You bet the code became production quality quickly. (I managed to screw up the AIX Smalltalk VM causing it to hang every couple of hours, no one was impressed).

Arguments I have faced include: We can't use our own product.

4) Simulated data. Make up data that you believe demonstrates that the system can handle all of your actual data. The idea is that the business changes rapidly, and meeting only the needs of current production data is not enough, you need to show that the system can handle more general cases.

This is what we are trying on our current project. It's later that I would like, but the business people are ready.

The Mythical Man Month

2006-09-06T10:36:00.000-06:00

Just reading this classic (far to late in my career). Excellent writing, wish I could write this clearly. He does an excellent job of describing the problems faced on a large project, and then addresses how those problems are solved by a practical waterfall approach, really how all the pieces fit together.

It would be really interesting to take his book and overlay it with an Agile approach.

One of the great quotes:

"On another project the best engineering manager I ever saw served often as
a giant flywheel, his inertia damping the fluctuations that came from market and
management people"

Discussion on coupling...

2006-09-05T16:45:00.000-06:00

This categorized how I try to work:

http://codebetter.com/blogs/david.hayden/archive/2006/08/26/Over_2D00_Architecting-Via-Pure-Fabrication-and-Indirection-to-Reduce-Coupling.aspx

Basically, decoupling is important when you need it. If you write objects well (single responsibility, encapsulation) you can decouple quickly through refactoring when demanded by the business or implementation.

Decoupling has a cost associated with it.

Teamwave review?

2006-08-30T20:01:00.000-06:00

A few years ago I helped write the conference manager portion of this product with a company called teamwave. Interesting to see the review. Happy to see the word robust.

http://www.eweek.com/article2/0,1895,2005008,00.asp

Stepping up to the plate

2006-08-28T20:55:00.000-06:00

Our teams are redefining the way we work together. As we enter a discovery phase in our project (what bugs remain?, what stories did we miss?) we're asking our teams to adapt and innovate.

Our goal is simple: Become production ready.

Our task is hard. We have a large complex system, and our customers need to see just how great the system is, and how quickly we can resolve problems. Our customers need to explore and discover the functionality available in the (huge) application they have been developing with us. We need to support their efforts. There are no simple processes to tell us what to do.

It will be interesting to see who on our development team steps up to the plate and changes our processes to help evaluate and stabilize the system. It is never easy to change the direction of a team, it is always interesting to watch those who lead the charge and see them flourish.

Is it you?

Great people and Great tools

2006-07-27T19:47:00.000-06:00

I've been working with Lu and John on our help system. It's been amazing to see what people can do when they discover a new tool to help them solve a problem and they are empowered to solve that problem.

A good (great) Alistair Cockburn article: (Courtesy of Joseph King)

http://alistair.cockburn.us/index.php/Characterizing_people_as_non-linear,_first-order_components_in_software_development

Usability Cobwebs

2006-07-26T20:02:00.000-06:00

Having fun with Lu and John on the project sorting out how to make our help more usable. Building paper mock ups, figuring out our users, identifying core tasks and generally having lots o fun.

Having done usability 20 years ago (Psych degree as well as Comp Sci degree) there a quite a few cob webs in my brain. Oh well, just figure it out as we go along.

Smalltalkers

2006-07-26T19:54:00.000-06:00

Having been reminded that I'm an old dude

http://www.josephmoore.net/2006/05/visual-studio-ruby-on-rails-and-old.html

Courtesy John Duimovichs Blog.

It's a different safety net...

2006-07-17T20:34:00.000-06:00

XP has a different safety net than traditional approaches.

Written spec -> functional tests
Written Design & Architecture -> refactoring & mentoring
Change Control -> Malleable code & visibility
Detailed writing ->oral communication
Everything thought out -> just in time
Look to the future and prepare -> look to the past and improve

Is this why combining Agile and Waterfall methods is the worst of all possible worlds? In both cases the safety nets are tightly woven (coupled?), one piece depends on all the others. Mixing and matching from both sides looses that "weaving".

Mistakes

2006-07-01T00:43:00.000-06:00

If you aren't making mistakes you aren't trying hard enough.

Speculative Solutions

2006-06-28T18:50:00.000-06:00

are good. Unfortunately they are not the end point.

There are two primary reasons I come up with a speculative solution to a problem:

1) I haven't encountered it before
2) I think we could be doing better than our current solution

I often need to remind myself that the speculative solution is not the answer, it it the starting point. It is a chance for me to learn and figure out if this is right. If it is the wrong answer I have two options:

1) Modify what I'm doing
2) Try something else

To do this I need to be in a position where I can change. If beginning my solution requires me to go to the end before I can evaluate whether or not it worked, I'm in big trouble.

Holidays

2006-06-23T21:50:00.000-06:00

Looking forward to some holidays. First couple of weeks in July I'll be in Ottawa. Some time at the inlaws farm, rent a cottage with my family and of course a little talk on Agile development at CGI Ottawa for my brothers group.

John Duimovich

2006-06-23T21:24:00.000-06:00

I worked for John back when computers were made of stone with little hampsters used to power them. He was one of the key people at OTI and really, really smart. I was one of the junior developers and really, really dumb.

John is one of the brains behind the IBMs (e.g. OTI) Smalltalk and Java VMs. He groks computers like no one I've ever met. His insight into how people program and how programs work is amazing. He's fun to be around as his mind works in weird and wonderful ways that make you "Wish I had said that".

Take for example his blog on "Visible ink"

http://duimovich.blogspot.com/

Nice to find his blog.

Blogger is pissing me off...

2006-06-11T17:36:00.001-06:00

Blogger has identified my blog as SPAM. Which probably isn't far from the truth, however...

It's really annoying that I have to enter some word verification everytime I want to post. I've asked them to turn it off, it's been more than a week (took me a while to figure out what had happened). It's annoying.

Vlad: On risk and performance

2006-06-11T17:36:00.000-06:00

Vlad (or Vald:) :
"Say there's a 99% chance that performance problems can be dealt with effectively late in a project, but there is 1% chance that performance problems can't be dealt with effectively. Is it worth risking an entire project on faith that one will find ways to fix the problems?"

Yes, on some projects performance is a severe risk. For example, my brother in Palo Alto needs to grab data from the partical accelerator as fast as possible and store it in a database (vast oversimplification). He worries about performance every second he programs. He writes custom ethernet drivers, uses customized hardware, reprograms switches and even writes in line assembly language in his C code. They spent 3 years making sure the software would be fast enough. Their risk was being able to do it fast enough, not whether or not they could store data in a database.

What is the risk on our project? Are we going to have to rewrite switches and introduce specialized hardware? Or just write clever java code? Our team is full of people who can write clever java code, it's not really a risk. I don't think it's even 999 to 1. Especially, when our code is trying make the business rules clear, and our tests form a living specification of the application. Acutally, I don't see a scenario where the talent on our team couldn't make our app performant, it's what we do and do really well. It is a technical problem, and we're really really really good at technical problems.

Our risk is the business and understanding how to make it work. Codifying the rules of production accounting is really really hard. Getting the specification right (the tests) and making it work correctly (the implementation) is a huge risk. How do we take a complicated body of (and not necessarily consistent) rules and make them into a reliable program? What are we missing that will make it hard to run properties? How does our model fail to meet the needs of our production accountants? We mitigate these risks by having production accountants test the application everyday. We write tests to create a living specification of what we've learned. We focus on these things, because if we get them wrong, we're in big trouble.

Incidentally, we have done back of the envelope calcs and shooting tracer bullets through the application to see where we need to be and actually are in terms of performance. In our heavy processing areas, our early calcs show we need to get 2-10x faster, depending on the area. Assuming, that is, we don't turn this into a batch process system, which would make us about 3x slower in the worst case. We can deal with that.

Vlad points out my oversimplification

2006-06-10T15:36:00.000-06:00

"Does everyone really say that performance tuning should be saved for last? "

Well ok, I meant deferred as long as possible, not dead last. People advocating this strategy include: Kent Beck, Martin Fowler, and I believe Ward Cunningham.

Martin Fowlers discussion
http://www.artima.com/intv/tunable.html

Kent Becks discussion
http://www.rolemodelsoftware.com/moreAboutUs/publications/articles/lazyopt.php

My experience:

On my last application we spent the majority of our time implementing the system and deferring performance. It took three developers six months to build. It was 10x too slow. It took 1 developer 3 weeks to rip out the guts of well factored code and put it back together fast enough.

Near the dawn of time, when I worked at OTI, there was a huge panic after IBM built Visual Age on top of OTI/Smalltalk because it was *WAY* too slow for them. One (Senior) developer, Steve, with the right tools(profiler and an open mind) went and assessed the situation. In a few hours things were all tickety boo. It was simply the way they were initializing their objects.

In my experience, with the code teams I've worked with, deferring performance saved time and money. It's the old 80/20 rule (80% of the time is spent in 20% of the code). In my experience, I think this is way closer to a 95/5 rule.

If you change (refactor) your code alot, that 5% will change alot too (it will be the heart of your system). How easy would allocation be to change if it had to meet performance criteria as well? How could we experiment?

Garbage Collection

2006-06-10T15:20:00.000-06:00

It's been close to 15 years since I worked with the garbage collector in OTIs Smalltalk. I was tasked with experimenting with parallelization for SMP boxes in OTI/Smalltalk. At the time, people were struggling with parallelizing garbage collections. The train algorithm was an interesting paper, suggesting better ways of parallelization.

I was simply curious to see if from a 50,000 foot view garbage collection has changed much in the last couple of decades. Apparently not.

This was a really good overview for me.

http://www-128.ibm.com/developerworks/java/library/j-jtp11253/

Empirical Performance

2006-06-10T10:17:00.000-06:00

Everyone tells you to do performance tuning last. Why?

Because you can make performance tuning empirical. If you've done your tests first you'll be able to change the code to be performant, without breaking it.

The following process works for me as follows:

0) Prioritize and Instrument

Record how long things take in your app. This can be as simple as paper and pencil, or as sophisticated as automated performance tests suites.

Look at what is slow. Where would you get your biggest bang for the buck? What would make your users the happiest? Where will the load be in your system (back of envelope math works well here)?

Are they having trouble with the GUI, or the calculations taking to long? Which page? Help them.

1) Investigation and Hypothesis

You need to understand what the application is doing and how it does it. Use a profiler such as JProfiler. Eclipse has a free but slow one (tptp).

Look at time spent in methods - can you reduce the time?
Look at who is allocating objects - can you reduce the number of objects?
Look at number of calls to a method - does it make sense?

Hypothesize where you can save time, have rough guess of proportion of time it will save.

Example: Alan and I found a method was called 20x more often than we expected. It was a bug in the code that was doing unnecessary calculations.

2) Experiment

Do actual timings on the code you are running (I use println like utilities). Youcan't use a profiler for this because of dialation effects (e.g. the OS and Database are not being slowed down, only your java code - the profiler may be lying to you).

Do back of envelope calculations to see if what you're doing makes sense.

Calculate: Theoretical maximum you can save, and likely amount you can save.

For example: assume you find a method you can double in speed. Wow that seems like alot of savings. So lets say this method is taking 10 seconds out of a 100 second process.

Now lets assume you speed it up so it takes zero time:

Theoretical maximum: 10/100 = 10%. Will that satisfy your users?
Likely savings (Double speed of method) = 5/100 = 5%. Will your users notice?

3) Conclusion

Is it worth doing?

If you know your users need a lot more speed 5% won't cut it, you'll have to go back to investigating and finding where to improve.

Otherwise implement the change and then measure your actual speed gain.

Empirical Performance improvements. I find them worth the wait.

Useful little tool

2006-06-10T09:48:00.000-06:00

For those of you who are part-time SQLers and need to format stuff occasionally.

http://www.wangz.net/cgi-bin/pp/gsqlparser/sqlpp/sqlformat.tpl

Service what you build

2006-06-10T09:42:00.000-06:00

I always try to help service what I build.

If you were to buy a car from toyota - you expect to be able to service it there. Same with a toaster.

This analogy only goes so far, because when someone builds a toaster, they usually build 1,000's (or millions). Servicing can be done by specially trained servicing staff.

When I develop software, my customers will need service. I provide that service.

Specification Based Factory

2006-06-06T22:37:00.000-06:00

Cool stuff today.

Simon and James came up with a new use for the Specification pattern. They combined it with the Factory pattern.

Basically, we need to find a class under some pretty heavy conditions (OK maybe we need a strategy pattern). They look up the class in a Map, indexed by a Specification object. They iterate over the map evaluating the specification, when one return trues they poof up a new instance of the class contained in the value.

Nice explicit code, with each rule contained in its own specification object.

I love simple stuff.

Process and people

2006-06-04T10:54:00.000-06:00

Had an interesting discussion with Rado the other day. We were discussing process.

Just reinforced the notion that process is there for the people, not the people for the process. As such the process should accommodate all people and their working styles, not the other way around. When running a project I must try and make the process we have chosen work for all the people on the team. The inability of a process to adapt to an individuals working style is a failure of the process.

In general I try to adapt the process to the people. In reality, at times I am not able to adapt the process to meet the needs of all the people. This is a failing of the process and myself.

Performance techniques

2006-06-03T15:10:00.000-06:00

Techniques we are using to improve performance:

1) Do more at once in a transaction.

With coarser grained transactions, we spend less time looking up stuff in caches and merging changes and more time actually doing the important work.

For example: Our importer was importing 1 measurement at a time. By grouping the measurements together on a per facility basis and updating them in a single transaction we reduced the time by 50%. DEspite caching

2) Cache.

Yep. It works. In our case we cache our snapshots (www.martinfowler.com/ap2/snapshot.html) which are constructed frequently in our networks of facilities.

For Example: Had one process come down from an hour to process to mere minutes.

3) Convert dynamically calculated results to stored results.

For example: Our statuses were being calculated all the time for complex large networks. Made the GUI slow. Store the results and poof things sped up by a factor of 5.

4) Convert stored information to dynamically calculated.

For example: We currently store some summary numbers for one of our facilities. Everytime something changes we redo our calculations. They end up being done over and over.

Instead, we make it an on demand calculation and our performance goes way up.

Many times we are pushing the time around, from a place where it hurts to one where it is OK.

Performance tuning in an agile world...

2006-06-03T14:37:00.000-06:00

I have taken over the lead on the performance tuning of our application.

A few interesting notes on performance tuning:

1) We do performance tuning test first.

We use Watir to drive our performance tests - because they reflect the experience of the user. We can measure the responsiveness of the app. Also we get to show the users what we are measureing and see if it matches their workflow in a month

On our project, we have moved away from the traditional pass/fail test for performance. We have red, yellow and green. Green means we're with in our target, yellow indicates we're within about 50% and red, well, we're in trouble. It helps prioritize our work.

As always we're running these tests a few time a day in a cruise-control like process.

2) There are different areas of performance tuning.

We have GUI response time, processing time and import/export time (our application has a large component of reporting and third party application interaction in it).

We concentrate on each one, writing tests and getting feedback from our testers on how the system is doing.

3) Break down the problem. Tackle one thing at a time to make it managable.

Performance, Load, Concurrency are all heavily interrealted problems. The nice thing is that we can takle them individually, we don't have to deal with them all at once.

We use both real and imaginary data to drive our performance tests (Actually, we use real data that are part of our regression suite).

4) Thank god that your developers have been refactoring code. You'll notice when you have some "original" design code, versus stuff that has been worked over. It is way easier to tune the latter.

Prescriptive vs. Descriptive

2006-05-16T21:52:00.000-06:00

I like descriptive writing. (e.g. Ron Jeffries/Martin Fowler)

Basically, they describe what they've done and the outcome.

I dislike prescriptive writing. (e.g. Scott Ambler).

They describe what you should do to be successful.

I read a bunch of my previous blogs. The ones I like are descriptive, the ones I dislike are prescriptive. I'll try to be more descriptive.

Help with problems not with solutions

2006-05-16T20:45:00.000-06:00

Are you leading an agile team?

One of my big (and hard) learnings has been that one of the key ingredients to a high performance team is how the leadership delegates responsibility.

I have found that helping people identify problems, and brainstorming with them (rather than for them) creates a great enabled team. It also creates a great relationship.

I have found that leadership for me, means letting the people I work for (my team) do what they think is best. They make great decisions. Any mistakes they make, they recover from. I like to help with advice, if they ask for it.

Here's a great article from the globe and mail on the subject: (sorry, you'll have to purchase it)
Caution: thinking encouraged here

Enthusiasm in an Interview

2006-05-06T12:18:00.000-06:00

Remember that enthusiasm is important for an interview. Why are you doing this job? Do you like building software? Are you learning new techniques? Are you going to excel on my team? Are you going to grow on my team?

These are all questions I'm trying to answer for myself during that 30 minutes I have to decide whether or not you'll work out on our team.

A big factor is how enthusiatic you are. I know if you love what you're doing, you'll be working at being better everyday.

For those of you that are counting...

2006-05-04T00:37:00.000-06:00

Our project has blown through 10,000 automated functional tests. Still another year to go.

170,000 lines of business code
230,000 lines of test code

Average # of lines per method: 5

# of great people working on the project: 62
# of customers 12
# of developers ~ 32
# of qa 5
# of teams 5

# of sprints: 24

Time to run 10,000 tests: 380 secs. TOO LONG.

On Performance

2006-04-30T21:58:00.000-06:00

Vlad commented:

You mentioned that performance could be improved with this approach, but I'm not
sure how - since this approach involves asking the database for all the records
and then filtering them usig the specs. Using toplink expressions, one can take
advantage of indexes on the database tables and the syntax is quite similar to
specs, so one gets the same kinds of benefits in terms of code duplication. I'd
love to see a posting about performance tuning these spec objects.

Performance for us was not over the database, but the number of times we were looping over the MeasurementPointCollection. Many of our common calls would loop over the measurement point collections two to three times. We saw about a 5% speed up in the tests using the Specification. Not a lot, but something.

Earlier in the project we used Home (Repository Pattern) calls to load the measurement points from the database. We found loading them once and reusing them much faster than using the database. (I think we halved the time the tests took). This is fairly common. (See Fowler PoEAA p.40, Toplink performance http://www.javaperformancetuning.com/news/interview030.shtml. In particular, read the question: What are the most common performance related mistakes that you have seen projects make when developing TopLink applications?)

Specification Objects

2006-04-30T11:52:00.000-06:00

Domain Driven Design by Eric Evans has several patterns mixed in it, among the ideas about how to leverage the domain in development.

One of the patterns is the Specification pattern. For me, it's primary benefit has been to reduce code duplication when you're using collections, and need to find subsets of the objects in those collections, depending on various Criteria.

For our project we have this problem in spades, with contracts and measurement points.

Sometimes we want to find all Sales contracts. Sometime we want to find an Oil Contract. Sometimes we want to find Oil contracts that are also Sales contracts. We have many many dimension on our contracts, and need to slice and dice them many different ways.

Our original implementation looked like this:

findAllSalesContracts
List results = new List
foreach contract in getContracts()
if contract.isSales
results.add(contract)
return results

findAllOilContracts
List results = new List
foreach contract in getContracts()
if contract.isOil
results.add(contract)
return results

findAllOilSalesContracts
List results = new List
foreach contract in findAllOilContracts()
if contract.isOil
results.add(contract)
return results

See the duplication, the repeated code? Imagine 4 or 5 conditions. This gets unmanagable.

Now here's a Specification Implementation:

class OilSpecification
isSatisfiedBy aContract
return aContract.isOil

class SalesSpecification
isSatisfiedBy aContract
return aContract.isSales

class ContractFinder
findAll(Collection, Specification)
List results = new List
foreach element in collection
if Specification.isSatisfiedBy(element)
results.add(element)
return results

findAllSalesContracts
return ContractFinder.findAll(getContracts, SalesSpecification)

findAllOilContractsList
return ContractFinder.findAll(getContracts, OilSpecification)

findAllOilSalesContracts
return ContractFinder.findAll(getContracts, SalesSpecification.and(OilSpecification))

Removed several hundred lines of code from our application.

Denis and ClearStream

2006-04-29T17:04:00.000-06:00

ClearStream was one of the companies I have seen where the words the "we're a family" was true.

Denis has taken care of his employees through thick and thin, and they have been good friends to him. When he spoke of them to me it was always with fondness and pride. Those who had left he always appeared to have hope they might return, while proud of their current successes.

My sympathies go out to Greg, Gerard, Jennitta, Ralph and Shaun who all form part of the ClearStream family.

Denis and Calgary

2006-04-29T10:20:00.000-06:00

A few years ago Tom and Mary Poppendick came to Calgary and in one of my conversations with them, they marvelled at what an agile community we had. It was unusual.

One of the reasons for that community is Denis. Denis created a business in new ideas and new ways of doing things. ClearStream is not a body shop, it is more. Denis' business sense made it possible to assemble some of the best people in the city and introduce innovative ways of developing on many different projects.

In the past he has helped his company introduce Smalltalk, Objective-C, object oriented development and now Agile development to the community. He brought the people together with the ability and turned it into a viable business. His companies innovation has pushed other bigger companies like IBM, Telus, CGI to adopt new techniques and practices faster than they normally would.

Denis helped form our agile community. Thanks.

Denis Clelland

2006-04-28T21:13:00.000-06:00

Denis was a great friend. If you knew him you very quickly thought the same thing.

My own relationship with Denis began as a business relationship. It very quickly became warm and friendly. Denis was genuinely interested in me personally. He did that with many people.

Denis was easy to trust. I could tell him lots and bounce ideas off him. He'd tell you about his experiences. Not "condescending" or "teaching", sharing. He was interested in my ideas, thoughts, family. He shared his with me. It was a friendship.

Friendship and business went hand in hand with Denis I don't think he could have one without the other.

I miss my friend deeply.

Drywalling...

2006-02-07T19:50:00.000-07:00

I've been drywalling for the last few months. I have lots of tips on how to mud, soon I'll be back to my normal posting schedule.

Verify imported data

2006-01-01T11:45:00.000-07:00

Just used Watir to verify some imported data.

The problem: We had business logic in java. It was much easier to do our imports in SQL. Our new logic had stricter controls than the old application in some areas.

How do we verify that our imported data is editable in the new system?

We wrote a simple script in watir to try editing every single piece of data. Nothing fancy, just open and save. The big trick is that you need a simple way to bring in the data easily. Our app allowed us to specify the surrogate key as a parameter to the editing webpage. From there watir hit the save button and then loaded the next piece of data.

Less than 50 lines of code. More than 24 hours to run. Found problems in .5% of the data.

Silly extremes...

2005-12-23T14:15:00.000-07:00

My last couple of posts have been about extremes. Rules that should be applied with a pound of salt.

Interestingly, they have had two interesting side effects. One, they make me think more about one aspect of how I'm coding. Two, they create interesting discussions with other people on my team about what the rule really means and how it has been applied.

Another piece of the "deeply nested if" smell...

2005-12-22T19:54:00.000-07:00

I was doing a little refactoring with Srini today and made a mental note of something to look for. When I see multiple nested ifs in a method, it seems to indicate a problem with responsibility. Usually, you are asking one object one thing, another object another thing, then even another object a third thing and then finally you operated on that third thing.

In this code it looked like you were reaching into the guts of other objects and playing with them. Once you reorganized the method, it was easy to move the ifs and their actions into different objects and the code began to make sense.

I'm going to play with a rule that says "I can only perform ifs on the results local methods/variables".

Lets see what happens. Anyone want to play with me?

From medium to fine grained...

2005-11-20T11:21:00.000-07:00

I'm trying to make the step from medium to fine grained objects.

1) Try to create an object with everything it needs at constructor time.
2) Try to have at most one public method to do the work in the class.
3) All other methods are simply for inspecting the state of the object.

I shouldn't have to call a setter on an object once I've built it. Otherwise I've introduced coupling.

What is your object level

2005-11-20T10:47:00.000-07:00

In the OO world there is much complaining about how everybody else is a bad OO developer. I'm beginning to understand how lacking my own skills are and am working on improving. However, I believe the vast majority of developers have not evolved their skills enough.

Working with a large team I get to see many of the different skill levels in OO that exist.

1 Modular developer.

Most of the code is organized along similarity lines. This is actually not bad. It can often be refactored into a set of parameter objects, method objects and data objects and which can then be reconstructed as first class objects.

Some smells are: Lots of static methods, Difficulty applying Gang of four patterns, Lots of parameters being passed to methods. Lots of ifs. Some apps look like PLSQL rather than OO.

2 Coarse Grained Object developer

There are a few key objects in the system. Often you'll see deep hierarchy trees with few branches. When you ask yourself the responsibility of an object you can come up with a list, or it is not clear.

Some smells are: it's a little harder to figure out where the behaviour for an object is. Often you'll find more coupling as two object call one another, this is due to the fact that they both have lots of responsibility. Lots of ifs.

3 Medium Grained Object developer

There are a large number of objects in the system, but it is still difficult to make changes sometimes. Each class has a few responsibilities. Not always alot of subclassing. Subclassing tends to override large portions of the super class.

Some smells are: incremental behaviour is hard to add and requires more work than you would think. Things feel good though.

4 Fine Grained Object developer

At most objects have two responsibilities, maybe one. You find that incremental change is really easy, adding a new way of doing something is straight forward.

Our code lives in all four levels. We have some modular code (Look at our test helpers) as well as some really clean objects (look at our allocation results). Most of our code varies between coarse grained objects and medium grained objects.

I think my skill level right now is certainly at the medium grained objects and I'm pushing myself to fine grained objects. It's fun and I'm learning stuff.

An important thing to remember for the OO community is that you can't go from modular to fine grained in a day. It takes years (decades?). YOu need to figure out how to make those steps incrementally.

I'm trying to apply patterns from Ken Becks Smalltalk Best Practice Patterns.

Embarrassing

2005-11-17T20:36:00.000-07:00

Homeless shelters over capacity

Alberta's surplus projected to climb to $8.7 billion

Make sure you give a lot more to charity this year. This is really stupid.

Requirements gatherer

2005-11-03T20:57:00.000-07:00

I think on my next project I'll endeavour to find a requirements gatherer. This is something that has worked out interestingly in our project and I think it is one of the missing pieces in XP.

When you have a customer they often lack the ability to express and prioritize their needs sprint by sprint. They need a grasp of the whole product. They need a grasp of what is important. They need to be able to help the customer prioritize to build a better product.

This is the person that needs to understand the big picture as well as the small. They also need to understand in their bones that development won't be able to build everything the customer believes is essential.

Watir

2005-11-02T20:11:00.000-07:00

We've been using watir on the project for a couple of weeks now. It is far superior to JWebUnit which we used previously. Great for testing your GUI througha browser. Ruby makes a good scripting language (I guess that's what it is supposed to be) and using the IE Ole Control as the browser engine for the testing harness is a stroke of brilliant simplicity. Stuff like JWebUnit tries to simulate browsers. Makes all those nasty "doesn't run the javascript right" problems go away.

You can find it here:

http://wtr.rubyforge.org/

Hah...

2005-11-02T19:09:00.000-07:00

Just spent some time explaining blocks to people who are writing tests in Ruby. To me, it is thank god their back (I started using then 15 years ago with Big Dave Thomas), but there is a whole generation who have never played with Smalltalk.

First thoughts on Ruby...

2005-10-26T20:45:00.000-06:00

Thank god blocks are back.

What the hell is up with all the special characters and weird syntaxes? Feels like C's question mark colon is coming back.
Why do people always think having multiple ways to do the same thing is a good idea?
Mixins are good because ...
Parser errors remind me of the LValue errors I used to get out of Microsoft C compiler.

Maybe static typing is good for code generation as it allows code completion and easier refactoring. (Did Ted really say that???)

Ruby on rails looks more like a steam train than a modern locomotive. Better than what we have, but not environmentally friendly.

Most importantly:

I've only used it for a couple of days. Ask me in a week.

A cripple without a cane...

2005-10-26T20:44:00.000-06:00

I'm doing a little work in Ruby. There are no refactoring tools or code completion.

The burn down to zero

2005-10-06T19:06:00.000-06:00

One of the things our project is missing is the excitement of completing what we think we can complete. Our executive team has declared that 60 stories a sprint is the goal for the team. Currently we're running at about 42.

Unfortunately, this means that the teams always feel that they are behind. There is no victory in completing all your stories, or even better if we did 50, we would still be behind. I think tomorrow we should pull 18 low priority stories off the wall. We can give our team a victory. They are a great team.

Dammit a good article...

2005-10-06T18:57:00.000-06:00

.. by Alistair Cockburn. When this guy is on, he writes really, really well.

http://alistair.cockburn.us/crystal/articles/aih/areiterationshazardous.htm

Good example of not getting caught up in the hype of XP. Remain focused on the end goal and make your development process work around that. Don't make the process the end goal.

Some problem solving strategies...

2005-10-06T18:21:00.000-06:00

When I'm trying to solve a hard problem, refactoring or otherwise I employ some strategies to help me get a better understanding and arrive at a good solution.

First I come up with a number of ideas. Very explicit different approaches to the problem. Stupid ideas are as welcome as ones I think are excellent. They make you think about the problem in different ways. I then cross ideas off the list that clearly are way too much work or won't work at all.

Of the ones that remain I'll choose two that look promising. If there is one that looks promising, but I don't like, I'll often try that one first. It might lead to new understanding. A spike of 1 - 2 hours will lead to new learnings and new strategies.

Sometimes, when I grow too attached to an idea, I'll hand the problem to someone else. I get enamoured with my own ideas, and this is a bad thing. By passing control to someone else I can be assured additional ideas will be generated and heard by the person working on the problem.

Is this double speak?

2005-10-05T20:40:00.000-06:00

http://www.amazon.com/exec/obidos/tg/detail/-/0596008740/002-0408435-3356841

A leadership role for software team leads

2005-10-01T10:28:00.000-06:00

On software development projects there can be lots of turmoil. We live in a time where programs can be extremely complex and difficult to estimate. This creates stress for business people who are trying to evaluate return on investment. Unfortunately, we cannot give guarantees - the technology doesn't allow it yet. This creates justifiable stress among the business people and turmoil on the team.

Software leaders have a different role to play than most other leaders. Yes, you're responsible for encouraging innovation, encouraging thinking about the problem differently to find newer and better ways to build things. The different thing in software development is that you're reponsible for protecting your team.

This means, you have to ride out turmoil and explain (not hide) it to your team members. Unreasonable demands from the business group must be tempered by your explanations and reasonable expectations. Unreasonable expectations from the business group must be fought visibly by you.

Above all you must be honest and not play games. Your integrity will serve you well.

Leadership is hard work everyday.

Innovations on an Agile Team

2005-09-30T23:58:00.000-06:00

Here are some of the innovations I've seen on our team: (Every time I think for 5 minutes I find 5 more - I apologize to all of you who have contributed more that I haven't listed)

Production Accounting Test Fixtures
Large FIT testing (3x)
In Memory testing
GUI Cruise tests
Distributed Cruise
Condor Tests
Opteron Tests
Java/Ruby-Watir Gui tests
Toplink prebuilt expressions
Separation of Master tests
Toplink mapping of Views
Automatic Db Validation
Pluggable Java Royalties calculations
Single GUI Dtos
Self calculating allocations
MyTestSuite automated generation
Date Effective Snapshot
MultiRow Importer for Excel
Value Objects
Monthly Measurements
Homes
Balancers
Initializers
Allocation Visitors
Xml Beans
Profiler
Database Façade
Form Helpers
Warning Handlers
Automatic Error Handling & Display
HSSQL for database tests
1 click Db generation
Test Profiling
GUI Profiling
5 ideas
Strategic Pairing
Team/Spoc Planning
Team Pods
Bug fix race chart
Burn up/down chart
Bug categorization chart
Refactoring Criteria
Macker

Be proud.

Colin Cassie

2005-09-23T22:45:00.000-06:00

A big thanks to Colin Cassie - our soon to be ex-project manager. He's on his way to retirement now, after giving a great boost to our project. His patience and experience were greatly appreciated by all. The way in which he allowed all those under him to work has made the project as great as it is.

The biggest lessons I've learned from Cassie (we all call him Caesar) is 1) to let those who work for you do what they do best, and 2) to stop those you work for from interfering with those who work for you.

Thanks Colin.

Individual Training

2005-09-23T22:26:00.000-06:00

How do you get people up to speed really quickly on a project? We have 26 developers on our project 8 of whom are absolutely new. How do we train them?

We offered our people (new and old) a range of training opportunities.

A couple of courses on the business
A bunch of courses on the app
Browsing Code Alone
Fixing bugs Alone
Pairing with a Newbie on a story
Pairing with a Senior on a Story
Pairing with a Senior on a refactoring
Pairing with Newbie on bugs
Working in a larger group to implement a story with Senior direction.

Which one got chosen? Well everyone had a different answer. Each chose a different combination. The best choice for training is to let the trainee tell you what they need.

Interestingly they all accepted the courses designed to describe the business and the implementations. There were 6 of them (1-2 hours each, 2/day).

Thanks to Colin Cassie for encouraging this direction.

Stupid Ideas that help me.

2005-09-22T22:24:00.000-06:00

This is funny. Having forgot my wifes birthday once, I might sign up. Oddly appealing.

Thanks Jeff.

Get it in production...

2005-09-21T21:06:00.000-06:00

Mostly when you're implementing a software system, you're replacing existing software. Be it a COBOL application, a pen and paper process, something people do on a calculator, or in a spreadsheet. Software usually isn't doing something new. Just replacing the old. This helps you prioritize.

Your goal is to be in production. Yesterday. On the first day of the project you should be PANICKING that you aren't in production yet. You're losing money everyday. Software in development has a present value of exactly $0.

This means really hard decisions when prioritizing stories. Here's some rules:

1) If you're rewriting a system - your new system shouldn't be able to do as much as your old system before going into production.

Your old system has a bunch of unused features. Ditch those. If you only use a feature in a couple of places, consider a work-around.

2) Your new system doesn't need to do anything new. Unless you're about to be sued, there are no risks in putting in a system that does exactly the same thing. And if you're about to be sued it has nothing to do with the software.

You're rewriting because the technology is outdated: meaning hard to update and buggy. See if the new system solves that problem first, before slapping all sorts of new features in it.

3) If you have a work-around OUTSIDE the system, keep it outside the system. You will lose almost nothing adding it to the system afterwards, if it holds up production - you're losing money.

4) Now look at what you've asked for. Find a way to just replace a piece of the system at a time. If you can break it into 3 or six months bits, you'll be a lot better off. Don't screw yourself.

Continue to panic about getting into production. If you ever stop panicking, you are screwed.

NOW YOU'RE IN PRODUCTION. You know a lot more.

You can evaluate whether or not you were successful in replacing your old system. Was it easier to get into production? Was it less buggy? Make your decision about whether or not to proceed.

WHEN you do go into production as a customer: INSIST INSIST INSIST that you see all the automated tests that cover the bugs you discover in acceptance testing. At least you won't have to worry about these during your next acceptance test.

If it's not less buggy and harder to get into production - decide whether to improve it or ditch it. Thank god you didn't waste a pile of money implementing all those new features.

NOW YOU'RE READY TO ADD A NEW FEATURE.

Start implementing the rest of the features. Make sure your next release is no more than six months away. Create as many of them as you can handle.

1) Implement all the things you are missing and everyone is bitching about.

Wow Feedback. Get it into production. If it's more than 6 months get it into multiple releases.

2) Implement your workarounds that you are currently using. You'll be solving real problems. Multiple frequency x average length of task. The higher the number the higher the priority.

Easy equation for prioritization. Only six months of features. No more. Get it into production.

3) Implement the new cool stuff.

You are now free to prioritize anyway you like, remember though you'll have some more of #2's popping up.

I won't lie, I won't lie, I won't lie, I won't lie, I won't lie, I won't lie

2005-09-21T20:38:00.000-06:00

One of the problems with Agile is the fact that :

Everyone likes a fixed-price, fixed-scope contract.

Unfortunately, as pointed out by anyone worth their salt in our field:

THIS IS NOT POSSIBLE on any project of significant size.

(Martin Fowler, Eric Evans, Kent Beck, Ron Jeffries, Mary and Tom Poppendieck etc...). You can try and wrap the package up anyway you like...list of features, fixed number of stories, written specifications. It will defy all attempts to put scope and features in the same box. Our field has not matured enough yet. It will probably be at least a hundred years, yet.

The huge downfall of Agile, is that it constantly tells the user, "no, we're not on the fixed priced/fixed scope schedule, we're behind". Our profession is full of people willing to say I can do X for Y dollars. Generally, they believe they are telling the truth. They aren't. On some projects the customer doesn't find out until the end. Predicting a software project scope and schedule is like trying to know the speed and location of an electron: physicists have proved it impossible.

I never want to lie to my customer. I want my customer as informed as I am. I tell them each and every month where I am. If they're disappointed, that is unfortunate, but at least they have the truth. The decisions they have to make EVERY ITERATION about scope and time ARE HARD. If you don't want to make those decisions, leave it to me and I'll make the best ones I can.

I promise you a couple of things people who say we can do it on schedule and on budget won't: I'll show you every iteration how much we got working and I won't lie to you, ever - even when it's painful.

Remember your needs

2005-09-17T11:51:00.000-06:00

Sometimes, things change in our life. Along the way we discover we need to make accomodation for new events, crisis, or on going things in our life. We are never the same person and we need to change our habits to accomodate these things.

How do we deal with this? The same way we deal with everything else. Try something. It may not work, it may require modification, but you'll learn something and probably come up with another better idea.

Even more important, is to communicate. Joe lost his dog a few weeks ago. It was painful and difficult for him to deal with. He discussed it with us and we were able to accomodate his needs.

Funny, the proudest moments of a leader are when they can accomodate someones needs. It means that what they do is about the people as well as the work.

Trout Fishing in America in Calgary

2005-09-14T20:57:00.000-06:00

Trout Fishing in America

If you have kids who are between the ages of 4 and 10, you'll love these guys. They are funny for adults and kids. Make sure you go to the Childrens concert on September 25th (NOT the adult one on the 24th).

Sunday 09/25 2005

You can get tickets at:

(403)288-2616 - http://www.calgaryarea.ca/bulletin/view_bbb.asp?MessageID=8464&Community=ss - All profits to be donated to the Alberta Children's Hospital.

You can find some of their music on the trout fishing in america radio. Click on

http://www.k-ampplayer.com/kampplayers/run_kamp.php?cID=1c2c82dd3c0a3ab

Look for the songs "18 wheels on a big rig" and "mine". Lots of fun. Lots of energy.

Letter to Culligan

2005-09-13T22:08:00.000-06:00

To Whom It May Concern,

Just thought you should know we have cancelled our service contract with Culligan. We signed up for the service where for $49 we would get a water cooler rental free and 4 water bottles to start.

We were assured that we would use three bottles a month for 22.95. Okay, sounds reasonable, as I've never used water bottles before.

We used two. We received three.

We used two. We received three. Kids are wondering why we need so much water.

We called and asked for two. Okay we were told. We received three. Kids are now walking around water bottles in our house.

We called and asked for two. No, we were told the contract says three. Great, that helps. We received three. Kids are now tripping over water bottles.

We called and said stop delivering water please. Apparently, now, we need to pay rent ($15) on the water cooler because we have no water. Right, what's in all these bottles? Kids agree it is water.

In fear of being forced out of the house by the ever accumulating water bottles, we called on August 30th and cancelled service. Cooler was picked up along with (full) bottles. Kids were relieved that we chose them over the water.

Received a bill Sept 1st for September rental of water cooler. I am unlikely to pay it. My account number is ...

Please, please, please deal with it.

Your perplexed, water logged ex-customer,

Ted O'Grady

Pushing the limits

2005-09-12T19:31:00.000-06:00

We're hitting the limits of patience, and really interruptability. Our tests despite the in memory stuff are now 10 minutes. (5400 tests). They do lots of stuff.

One of our guys is implementing Grid unit. Fortunately, developers spend lots of time thinking leaving lots of spare CPU cycles lying around. Combined with a dual cored - dual Opteron box we expect our test time to go sub 1 minute.

Long tests have many bad impacts:
1) Long queues to check in (large team)
2) Disruption in thought process
3) Disruption in momentum

Thanks to Vlad, Zach and Alex and all the others that have provided different seeds on the team to thinking about how to speed up the tests.

Working with India

2005-09-08T18:51:00.000-06:00

Going to be working with an Indian team on our project. Should be interesting. Need to figure out all sorts of stuff. Need to email Jane, get the in the trenches view of things. She is doing it with thoughtworks. If there is anyone who can figure this sort of stuff out and make it work, it is Jane.

How do we measure progress?

2005-09-08T18:36:00.000-06:00

One of the things waterfall projects have over XP projects is Gantt charts. In the middle of the project they give your customer confidence that you are proceeding according to schedule. they put off the pain until the end when the charts need to be extended by testing and such.

In XP we can only tell them what we did. Customers want a measure that tells them you did as much this month as you did last month and how much is left and how long it will take. Velocity measures yesterdays weather. On other projects we have capped the budget, and report what we think the last story will be.

On this project we want to cap the time, complete a specified number of stories (2400), and budget is less of an issue. We don't do design up front - we do incremental requirements (stories). The last couple of sprints, our velcoity has decreased. In our estimation the stories have gotten bigger (have more pieces to them). What can we use to be honest with ourselves? Have our stories gotten bigger? Or has our code become more complex and we've slowed down? How do we measure this so we don't lie to ourselves? How do we measure this so we can explain to the customer?

Where do I sort?

2005-09-08T18:23:00.000-06:00

Sometimes there are questions about where sorting should occur.

1) Should I sort in the database?
2) Should I sort in the business layer?
3) Should I sort in the display layer?

The time you need to sort - is when you need a sorted collection. You cannot rely on someone else having given you a collection in the correct order, thsi tends to make thing hard to understand and refactor.

For example, if you sort in the database - how does your display layer have any knowledge of how things are sorted? How does it know the sort order won't change? The safest thing to do is sort at the point where it is obvious you need to sort. Sometimes that will be in the business logic, other times in teh database, and other times in the display layer.

But, you say, I might create a performance problem if I sort too often!

Wait until it's a problem - it probably won't matter.

We are connected

2005-09-02T17:28:00.000-06:00

I live in Calgary, Alberta, Canada. It's a city of a million people. I purchase my domain hosting from directnic. I used to have no idea where they were located. They have provided excellent service, and I have never needed to actually talk to them.

Now I know they are located in New Orleans. I hope they and their families have escaped the carnage and thank them for their service. The best thing I can do to help is be a patient customer. I will be there when they're ready. We're all connected.

Sonexis Raises Money

2005-08-26T22:16:00.000-06:00

http://boston.bizjournals.com/boston/stories/2005/08/01/daily10.html

Interesting. Software I helped build a few years ago.

Developing software is a force of will...

2005-07-27T08:03:00.000-06:00

Software gets complex whether you want it to or not. There are no magic bullets when it comes to developing software, following the right process or doing the right steps guarantees you nothing.

There are a couple of things that I've seen on all my successful software projects:

1) Force of will - willingness to do what it takes to make stuff work
2) Self discipline - trying to keep yourself from making the problem more complex than it is

These are things that come for free with good people.

Holidays!

2005-07-25T20:58:00.000-06:00

I'll be in Ontario/Quebec in August. Three weeks. Looking forward to spinning down, maybe do a little exploratory coding. Should see what Mark is up to...find some place in the middle.

More Developers

2005-07-25T20:57:00.000-06:00

We're hiring again. Another team, more developers.

Interviewing is keeping me busy.

The Power of 5 Ideas

2005-07-20T21:43:00.000-06:00

We have asked our developers to come up with multiple ideas for each problem they're trying to solve. (One of Eric Evans recommendations). We were having the problem where we found one solution and made that work, rather than finding multiple solutions and selecting the best. To start we've asked for five. Bad ideas are as welcome as good ideas. It has had some interesting side effects.

Some observations of Zachs team:

1) Coming up with five ideas requires both partners in a pair to work together to come up with ideas. This leads to cooperation when creating ideas, more willingness to listen to other peoples ideas.

2) When coding, the idea seems to belong to both of you. It is no longer one persons idea, the act of writing them down together makes them everyones ideas. Coding can become more cooperative.

3) Just generating ideas makes you more open to change, and less tied to any single idea.

Nice unexpected benefits of introducing another good habit.

Behaviour or test driven development?

2005-07-18T19:28:00.000-06:00

Dave Astels thinks it is important to change our idea of test driven to behaviour driven development. http://daveastels.com/index.php?p=5

Brian Marick thinks we should ask users to provide examples rather than tests.

Gerard Meszaros writes Unit tests to reveal object behaviour.

We write Functional tests rather than Design tests. It makes refactoring easier.

Which is right?

Well all of them provide focus on different aspects of the testing/behaviour of the system. They are all revealing the importance of different aspects of test driven development. Why do we need to choose one? Really, we don't. At different times each of them will become important.

Sometimes I want to describe the behaiour of the system. Sometime I want an seminal example of the system. Sometimes I want to drive my implementation. Sometimes I want a regression suite. They are all important at different times.

Found Dave Astels again...

2005-07-18T19:19:00.000-06:00

Added his latest blog to my blog roll.

Mentoring

2005-07-05T20:31:00.000-06:00

Mentoring is always a fun mix of:

1) Explaining
2) Letting someone do on their own
3) Demonstrating

It's especially fun to mentor someone whose bright.

Theories and Experimentation are like ...

2005-07-05T20:27:00.000-06:00

Models and Coding.

Many models are good, they let you think of thing in different ways. They don't have any answers - only questions.

Coding is an experiment to see if a model works out. Some are right and some are wrong. And some are good enough for our current understanding.

Don't get hung up on your model (theory) or your code (experiment).

A Productive Team

2005-07-03T20:57:00.000-06:00

The team I'm currently working on is highly productive. They motor through stories on a regular basis, there is little waste. They have a lot to be proud of. It makes my job easy.

On the other side the lack of waste also means a lack of slack - time to think and reflect on what you are doing. In the last week we allowed the team to "innovate" - take four days and do anything but stories. After being heads down for so long this week was a welcome relief, but also a huge learning experience for them. The payoffs for the project will be huge. People refactored, explored new ideas, and learned on their own for over 4 days.

In the past we have said to the team that they have at least four hours a week to innovate, that they should apply that however they can. The decision to defer a story for learning time is a hard one for a hard driving team and they haven't been able to make use of that time.

We will now be asking them to figure out how "innovate" like we did this week. They'll need to figure out strategies to be able to do it while developing stories. And I know we'll only gain productivity.

Tests

2005-07-03T20:56:00.000-06:00

For those who are counting we now have 4700 tests running.

Problems and Solutions

2005-06-27T21:59:00.000-06:00

Problems finding solutions.
Solutions finding problems.

Both are valid ways to uncover new ground. But remember which you are doing. If you have a problem and are looking for a solution - explore many solutions. You want to understand the best fit for this problem.

If you are a solution looking for a problem - explore many problems. You are looking to see if this solution solves any of the problems you currently have.

Business is Personal

2005-06-25T08:55:00.000-06:00

I often hear the expression "It's nothing personal it's just business".

Really? In our culture we tie our jobs very closely with our personal life.

Our business relationships are built on trust. We are dedicated to our work. We work well in evironments that are fun and friendly. When work goes badly we are stressed. Most people I know think about work in their spare time, trying to make it better.

We live, and expect people to move, close to work. We expect people to spend 40 hours a week with their coworkers. Many companies try to create an atmosphere where work is another family.

Business is personal. People who are laid off (for financial reasons) take it personally. Why did you choose me instead of them?

As a leader you must understand business is personal. The decisions you make affect people personally. Although you must make them for business reasons, you must realize the personal impact.

Refinement

2005-06-17T22:08:00.000-06:00

Courtesy Colin Cassies and Colin Jones:

The reason we revisit code is to refine it.

This is the same you would do with a word document, or a powerpoint presentation. The first time you write your words on a page, they lack form and often substance. They do not flow together well. You write and rewrite until the points you make are clear, concise and meaningful.

Welcome to refactoring.

Strategy 8 - can't move anything

2005-06-07T16:01:00.000-06:00

You don't like how the code looks. Your methods are eight lines long, but they only call methods in your current object. Moving those methods is not easy because they reference the current object.

Refactor by Inlining. You need to find different ways to break up the methods, so that dependency on the current object is broken. Once you've inlined everything, try moving stuff around grouping it different ways. Then extract methods and see if you can move them elsewhere.

Strategy 7 - Method longer than 10 lines

2005-06-07T14:35:00.000-06:00

If your method is longer than 10 lines you might be on the verge of being procedural.

Try using the extract/compose method refactoring (basically your method indentation should be a most one deep). Then look to see if those pieces you removed could be moved elsewhere.

Strategy 6 - Still too many ifs

2005-06-06T20:36:00.000-06:00

You tried moving stuff around and still ended up with ifs.

Refactor the ifs into the setup - you may find a factory allows you to create the appropriate object for the appropriate if.

Strategy 5. Make your code sound like the business

2005-06-06T19:51:00.000-06:00

If your code sounds nothing like the business problem you're implementing you're going to have trouble refactoring. You have implementation concepts mixed with business concepts.

Convert your code to sound like the business and your changes will be a little bit easier.

Strategy 4 - Too many loops

2005-06-06T17:53:00.000-06:00

If there are too many loops in your code the responsibility is in the wrong place.

Refactor. Each method should have about 1 loop. I find drawing a sequence diagram here quite useful. Usually you need to inject another object or push the inner loops elsewhere.

Strategy 3 - A bunch of Ifs

2005-06-06T17:51:00.000-06:00

If you see too many ifs in your code and if you can't understand it and if the ifs are used all the time (you get my point)...

Refactor. Maybe subclassing or delegation will help. Look at what the ifs are operating on - if they are on another object you probably need to move responsibility to another object.

Strategy 2 - Code doesn't reflect conversations

2005-06-06T17:50:00.000-06:00

If your code and your conversation about how things work are different...

Refactor. The code is trying to do stuff in a different way. Probably it has evolved from a point where it described exactly what the user specified. Along the way special cases were added and more functionality. Now that your understanding has evolved so can your code.

Strategy 1 - Lots of bugs

2005-06-06T17:49:00.000-06:00

Testing finding lots of bugs in your code?

Refactor for clarity. Look at the method names. Look for ifs and loops as places to refactor. Look at responsibility - is the guy with the data doing the work, or is someone else?

Sophisticated Software Development

2005-06-06T17:26:00.000-06:00

When I'm out, it is nice to be sophisticated every once in a while. However, in general I'm an ordinary guy with a wife, two kids and a dog who enjoys some terrible TV shows.

My coding is about the same. Sometimes I can write really, really nice code. Small, tight and sophisticated. A bunch of the time I'm just an ordinary guy. Being sophisticated is work, and really, like most people I can't (No matter how much I try) pull it off all the time. Yes, duplicate code is bad. But I think 2 duplications is better than 3 duplications. I'll try harder on the third duplication to make them all the same than the second.

Sophistication is a sign of maturity. Most of the code I write is new and hot off the press. Making it sophisticated is really hard when it's new. My code will get more sophisticated (read less duplication, more intention revealing) as it gets older and more experienced (ok, more used, refactored).

Sometimes I think the XP community is just trying to move the problem around from: "you need to be really smart to design" (waterfall) to "you need to be really smart to code" (XP - no duplication, write intention revealing code). Agile will fail if all the projects need the top 10% of developers. Agile must work for ordinary development teams.

I think we need to stop telling ordinary developers they need to get smarter about writing code and start showing them how to help their code get sophisticated as it gets older. Strategies for figuring out how and when to refactor AND redesign AND where to move the responsibility will help a lot more than asking them to get it "right".

Your code should mature as it ages. Maybe I'll write down my strategies...

Smaller objects

2005-06-05T13:07:00.000-06:00

We're getting rid of a bunch of the visitors in one part of our code. It is interesting to see how a pattern that is extremely useful and concise in an early stage of a project can become a lead weight around your neck later in the project.

Additionally, the FIT test are also coming into play as we make these deeper refactorings. They point out change that our functional test are missing. The integration of FIT throws up automated warning bells as our refactoring has unexpected side effects. Generally, fixing these mistakes are relatively easy, since our functional tests catch the big errors.

Finally, the code is becoming more distributed among the object, responsibilities shifted to the people who control the data. All in all, its fun to do and eliminates hundresd of lines of complex code.