markdietz

my blog

Software re-writes

This is a good post on how to improve the chances of success when doing a major re-write of software:
http://onstartups.com/tabid/3339/bid/97052/Screw-You-Joel-Spolsky-We-re-Rewriting-It-From-Scratch.aspx

Some thoughts to add:

Don’t try to add new requirements during the re-write

The functionality of the existing system is often not exactly what the users and product owners want. A re-write gives them an excuse to fix all the past wrongs. There’s just one problem: trying to re-write and change the functionality in the same step is orders of magnitude harder than replicating the existing functionality, testing that it worked, and only then changing functionality later. This is for two reasons: the users are going to think any differences mean the new system is wrong and if the existing functionality is replicated then much of the testing can be automated. Computers are good at detecting differences. They are bad at knowing if those differences are good or bad. If it does turn out a difference is due to a bug in the old system and replicating the behavior would be costly, consider fixing the old system so that the output going forward will match. For example, if something is failing due to an integer overflow, it is probably best to fix it rather than propagate it to the new system. However, if the difference is that the old system runs in daily batches and so takes 24 hours to propagate, don’t try to make the new system real-time before first replicating the batch functionality.

The existing system is an executable spec. Use it!

Have you ever wished the product team could give you an executable spec that was defined so precisely a computer could understand it? That’s exactly what the old system is. It may be ugly and hard to understand, but it is more precise than any set of requirements could ever be. A product person or business analyst might want to write a spec for the new system. Instead, if there is any question about how something should work, don’t ask how it should work, consult the existing system.

Who wrote the first system?

One of the more common times for a re-write in my experience is an MS Access or MS Excel application written by a non-developer domain expert has outgrown its original purpose, become critical to the business, and now needs to be re-written as a “real” application by the IT organization. There are many legitimate reasons that this is a good idea, but beware thinking that since the first app was written by non-developer so now that you’re doing it “right” with professional developers then it will be easy. Can you learn his/her domain faster than he/she can learn to program? Is it better to have 20 years understanding the business and 1 year teaching yourself to program or 10 years learning to program and 1 month learning the domain? The answer depends on the people, but it is definitely not an obvious win for the programmer. Often times the hardest part about implementing a system well is understanding the business domain. During the re-write, the most important person to have committed to the project is that original author. Remember, that person understands the domain so well they can teach it to a computer. The original system may not be sufficient to support the business going forward, but it is good enough that it is supporting a business valuable enough to justify investing in a new system. If the original author is not committed or has already been burned by a previous failed re-write, try really hard to get them on your side.

The existing system can be changed

Change the old system if it helps. Yes, even if it is an MS Access application. Until you turn the old system off, there isn’t really an old system and a new system. There is one giant system with multiple languages and technology stacks and redundant code. It might help to add some new logging to the existing system to allow automated comparisons between the old system and new system behavior. Or a new data snapshot function that can be used to help tests migration scripts. If you treat the old system as that “icky thing written in a language I have no interest looking at” then you are doing yourself a disservice.

Summary

There are a lot of things that can go wrong during a software re-write, but it has some unique aspects that can be used to your advantage. If you have a project like this, you’ll need every advantage you can get. Good luck!

Advertisements

Distance Measurements

Have you ever thought about how distance is calculated? Interestingly, there isn’t just one answer that applies in every situation.

If you have an x-y coordinate system, the normal method for calculating distance from the origin to point (x, y) is Euclidean distance

dEuclidean = √(x2 + y2)

This is derived from the Pythagorean theorem. It is probably what most people think of when they think of distance.

Probably the next most common distance calculation is Manhattan distance:

dManhattan = x + y

This is how you calculate how many blocks you would have to walk to get somewhere in Manhattan. The difference is that in Euclidean distance, such as a wide open field, you can cut diagonally across the grid. However, in Manhattan there are buildings in the way so all movement must be made across the grid. A point one block east and one block north (1 + 1 = 2) is just as far away as a point two blocks east (2 + 0 = 2).

An interesting pattern emerges if you write these two distance formulas a little differently. Note √x = x1/2 and x = x1. So:

dManhattan = (x1 + y1)1/1 = d1

dEuclidean = (x2 + y2)1/2 = d2

This might lead you to ask what does the general case dp = (xp + yp)1/p look like for different values of p. I only know of one other special case that actually gets much use. It is one you might not expect but have used if you’ve ever played chess. A king in chess is allowed to move a distance of 1 as defined by Chebyshev distance, which is:

limit p → ∞     dp = (xp + yp)1/p = max(x, y)

What this is saying is that not only can you cut across diagonals but you only count the maximum distance traveled in either direction. A point one block east and one block north is considered only one block away. In chess, the king can move one square in any direction including diagonal. The Chebyshev distance between two squares on a chess board is the number of moves it would take a king to move from one square to the other.

One thing I find interesting about these is how they relate to board games played on a standard square grid. Many games allow moving a certain number of squares on a board. Some board games do not allow diagonal moves. Moving from a one square to a square diagonal costs 2 moves. These games use Manhattan distance. Dungeons and Dragons editions up to 3.5 use an approximation of Euclidean distance where the first diagonal taken costs 1, the second diagoal costs 2, and the third costs 1 again. Other games do allow moving along diagonals at no extra charge such as Dungeons and Dragons 4th edition.

Note the generalization dp = (xp + yp)1/p is called Minkowski distance. Here’s an interesting graphic from Wikipedia of the unit circle – all points of distance 1 from the origin – for different values of p.

Minkowski3

An example of why I like scala’s syntax

The code below is a client for an HTTP webservice. It is an example of why I am really enjoying working in scala after several years of working mostly in java. The example is to show how concise the scala code is compared to equivalent java code. I’d love to hear any comments on how either the scala or java code itself can be improved. Here’s the scala code:

  // method definition
  def event(userId: String, event: String, timestamp: Option[Long] = None, properties: List[(String, String)] = Nil) {
    val params = ("_u", userId) :: ("_e", event) :: timestamp.map(t => ("_t", t.toString)).toList ::: properties
    val response = http.get("?" + params.map(p => p._1 + "=" + p._2).mkString("&"));
    if (response.code != 200) {
      throw new RuntimeException(response)
    }
  }
  // client invocation with timestamp and properties
  service.event("user_id", "my_event", Some(123456789), List(("prop1", "val1"), ("prop2", "val2"), ("prop3", "val3")))
  // client invocation with no timestamp or properties
  service.event("user_id", "my_event")

Here’s the equivalent Java:

  // method definition
  public void event(String userId, String event, Long timestamp, Map properties) {
    StringBuilder paramString = new StringBuilder("?_u" + userId + "&_e" + event + timestamp == null ? "" : ("&_t=" + timestamp));
    for (Map.Entry property : properties) {
      paramString.append("&" + property.getKey() + "=" + property.getValue());
    }
    Response response = http.get(paramString.toString());
    if (response.code != 200) {
      throw new RuntimeException(response);
    }
  }
  // client invocation with timestamp and properties
  Map params = new LinkedHashMap();
  params.add("prop1", "val1");
  params.add("prop2", "val2");
  params.add("prop3", "val3");
  service.event("user_id", "my_event", 123456789L, params));
  // client invocation with no timestamp or properties
  service.event("user_id", "my_event", null, Collections.emptyMap()));

The method body is 5 lines of scala versus 8 lines in Java even with using the ternary operator. I also really like how in scala we simply add all the query parameter tuples to a list and the join them to a string in a single call. Another nice thing is that the scala method signature indicates which parameters are required and which are optional. It is possible in scala to pass null to a parameter even when it is not an Option, but if Option is used consistently in your APIs then it can be a pretty powerful convention. It’s a hell of a lot better than javadocs!

The big win is on the invocation site. Scala’s tuples and List intializer let you simply inline the parameters. Scala is 1 line regardless of how many parameters. Java is two lines plus one for each query parameter.

The expression

timestamp.map(t => ("_t", t.toString)).toList

probably bears some explanation. An Option can be treated like a collection of size 0 or 1. The .map method returns a None if timestamp is None or returns a Some((“_t”, timestamp.toString)) if timestamp has a value. Basically this is functioning as a null check. Then since we want to append the result to a List, we just use the toList to make it an empty list or a List with a single tuple.

Obviously this does not even touch on really any of scala’s more advanced features. I think it does give an example of how nice scala’s syntax is even when writing pretty straight forward java-style code.