Monday, November 20, 2006

Thinking from Memory Defragmentation to Architecture Reengineering

Recently I don't know if its just my luck or the whole industry is planning to upgrade their architecture, almost every friend I talked to and even every interview I had asked me the same question “How to drastically improve your architecture without trashing the existing one”, and my recent research and study in a seemingly irrelevant field “Memory Management” provided me some unexpected insight into this topic.

Despite the apparent disparity between these two topic, there are some shocking similarity under the skin for these two. Lets take a look at a classic problem in Memory Management domain the Memory Fragmentation problem. In short, Memory Fragmentation is a memory state, usually caused by some long-running process, in which blocks of allocated memory scattered between blocks of free memory instead of one continuous extent. Memory Fragmentation causes the performance of memory allocation to decrease or even prevent program from allocating memory if the free memory is too fragmented. Memory Defragmentation is a reverse process that we use to fix the fragmentation problem, and the standard approach is pretty drastic: we can shut everything down (or at least freeze everything), free all memory, and restart (or resume) the whole program.

Now how about the architecture side? Although you usually will not have fragmentation problem in your architecture (notice I said usually), but without careful design, relentless refactory, and proper migration path software architecture tend to get outdated pretty quickly (what I call Architecture Deterioration), if you had experience integrating with some legacy systems out there you will know what I mean, and that's why we have the Architecture Reengineering to basically reverse the architecture deterioration to incorporate better technology and better approach before the architecture becomes another “legacy system”. The standard approach of Architecture Reengineering could be pretty drastic too: we can stop releasing new features or updates for the product or solution where the architecture was used for 6-12 months, redesign and implement the architecture and make sure it fulfill all the requirement that the predecessor does, then resume the new feature and release cycle.

Do you see the resemblance now? As you can see that both of the standard approaches to these two problems are probably not good enough for most of the situations we as software engineers commonly encounter. The standard Memory Defragmentation approach will result at least a long noticeable freeze which will lead to not only performance lose but even some unacceptable consequences in a mission-critical hard real-time system. On the other hand, the standard Architecture Reengineering approach won't even work in most of cases simply because freezing feature release cycle for 6-12 months will simply be committing suicide in the business world. So what is the better way to attack this kind of problem? Simple, instead of try to fix the whole thing in one shot, take an incremental approach fix the system in bits and pieces. First, lets take a look at how people perform Memory Defragmentation in an incremental fashion. In a normal C-based system, this can only be done with the classic programmer trick

Any problem in computer science can be solved with another layer of indirection. -- David Wheeler

If programs are not allowed to create direct pointer to the memory address, but only pointers to pointers that are pointing to memory, defragmentation process can move memory and only update the single pointer in the system, hence incrementally defragment the memory space.

Following the same line of thinking, adding a another layer of indirection should be able to help us reengineering existing architecture. Here I want to introduce a design pattern that I often use to help me to perform incremental Architecture Reengineering and eventually achieve what I call Continuous Evolving Architecture. This design pattern consists of two basic steps:

First – You need to encapsulate the part of the system you want to reengineer to level-4 which is the service level (or just wrap the whole system if you are planning to do an overhaul on the entire architecture). When I mentioned service here, I did not mean you need to implement a set of Web-Service interfaces and use SOAP as the protocol, but a rather more conceptual idea of services.

Second – You need to implement a classic Message BUS as a mediator and proxy to expose the services that you just created for your existing system through pure message exchange. If you are familiar with the Enterprise Service BUS concept, you can imagine this as a small scale implementation without all the Web-Service and BPEL stuff.

Now you have a layer of indirection that allow you to implement new features that might heavily rely on the existing system but without the need of talking to the existing system directly or even knowing the existence of the existing components. At the same time, it gives you the freedom to change any existing part of the system according to your own time line and budget without worrying about affecting the rest of the system, and hence a path to reegineer your existing architecture incrementally and safely.

Thursday, November 02, 2006

Two common mistakes that people makes when adopting Scrum for the first time

I came back from an almost a-month-long vacation about 2 days ago fully refreshed and with somewhat terrible jet lag as well, but the worst thing is since I currently don't really have a full time job yet therefore there is no motivation whatsoever for me to adjust my biological clock this time that's why I think my timezone right now is still somewhere around Honolulu.

In this first post of November, I want to talk about two common but sometime fatal mistakes that people usually make when they attempt to adopt Scrum methodology for the first time in their organization.

Adopt Scrum as everything but also the only thing

A lot of people adopt Scrum without realizing that Scrum is not a complete package. As stated on Control Chaos website:

Scrum is an agile, lightweight process that can be used to manage and control software and product development using iterative, incremental practices. Wrapping existing engineering practices, including Extreme Programming and RUP, Scrum generates the benefits of agile development with the advantages of a simple implementation.

As you can see, Scrum by itself is not complete because it is only a management process including a set of principals, disciplines, and artifacts that can be used to manage a agile and iterative project, but it does not contain any engineering practices thats why it must be co-implemented with other compatible engineering practices, such as XP and RUP, in order to be effective. For people who forgets about this or misunderstands what Scrum is all about, can turn a project into total chaos instead of controlling it, since Scrum alone does not address the following major engineering concerns:

  • Design – How and when the architecture is drawn and the major design choices are made?

  • Risk Management – How and when the technical risk can be mitigated?

  • Quality Control – How, when and who does the job to guarantee the product meeting customer's requirement as well as the quality expectation?

I can hardly imagine any project that fails to address these concerns delivering high quality product on-time without blowing off budget.

Chicken-head syndrome

I called it chicken-head syndrome when you see a Scrum project where the “chickens” are making decisions or even creating and dispatching ad-hoc tasks to the “pigs”. This usually happens with inexperienced Scrum master who allows “chicken” to give orders during the daily Scrum meeting instead of being a mere observer, since in the most of organizations at least some “chickens” are usually higher up on the organizational chart. A mild symptom of this syndrome is seeing “chicken” jump in during the daily meeting or initiating too many discussions or meetings with the developers (pigs). In a Scrum project a good Scrum master should act as a umbrella that shields the team from unnecessary communications from the outside world (personally I prefer 0 external communication during an active Sprint) hence allowing the developers to be productive and focus on completing their backlog items and constructive communications with the product owner thats why an iteration in Scrum is called Sprint not socializing or mingling. Projects suffer from this syndrome usually result in drop of productivity which eventually leads to schedule overrun, budget overrun, or both.