Jake Herringbone

Should users edit data in serialized text files?

Let me begin with an allegory.


❝We work at a great software shop. We are tasked with writing a large, complex piece of software. The domain for this app includes a rich, hierarchical data structure, with a ton of cross-references between elements, and our user will need to manage all these data elements. Furthermore, we will have a lot of users on the system at once, and they will frequently work on overlapping parts of the domain model concurrently, as well as browsing the history of actions taken and how they affect the model. We expect our users to be mostly “advanced”, so we can show them a complex view of this complex domain, and present them with some power features. Our basic architecture will be a speedy user interface, which the user will spend a great deal of time in, and a server which stores the domain model elements.

This looks like a pretty hefty design challenge. Let’s say one guy on your team (Sloth) proposes the following:

We can store our complex data model by picking one level in the hierarchy, let’s say the parents of most of the leaves, and writing it to a file. The rest of the hierarchy will be roughly mapped to directories containing these files. Inside the file, we’ll start with a bit of metadata, listing all the cross-references to the parts of the domain outside the file. We’ll come up with a nifty grammar so that we can serialize into each file in a roughly human-readable format. [ed. Sloth has been reading about domain-specific languages lately and has been waiting for a chance to engineer one]. To provide long-term persistence over server restarts, and also to handle the versioning and history requirement, we’ll use a Version Control System like Mercurial, and save each new revision of the file there using an account owned by the user. So that’s how we’ll store the data.

The rest of the team is mostly nodding to this point, they especially like his idea to use a VCS to handle persistence and versioning. One engineer points out that this means the versioning and history will only be available per-file, so they can’t version the leaves of the domain model. Everyone nods uncertainly, since that requirement wasn’t clear. Another engineer says he’s worried about the concurrent editing, if there are conflicts between users, they’ll have to merge that serialized file, which will happen more if it contains a larger chunk of the model. He asks Sloth,

“Why do you want to store a subtree of depth 2 in the serialized file? We don’t want the user to have to edit those files by hand in case of a conflict, right?”

Sloth gets a little defensive, and replies, “Oh, I thought they would edit them by hand all the time. The UI can just show a master-detail view with the directory structure and the contents of the selected file. Then we won’t even need a server!”

Everyone gets very uncomfortable. The tech lead feels it’s his job to lead Sloth a little.

“In other programs I’ve worked on of this complexity, we don’t show the user the data model in such a raw form. We want to separate the metadata out, especially all those cross-links at the top. And what will we do if we want to change the presentation a little? We’ll be stuck with whatever is in our serialized data format.”

Another engineer pipes up. “Yeah, this data model is looking really complicated, I don’t see why we’d make the user hand-edit the data. Maybe if we were just a couple guys and didn’t have time to write a server, we’d have to do that. What about avoiding conflicts? There’s no way the users will be able to keep those files consistent when they make changes.”

The database engineer is feeling a little left out by the proposed file format, since it puts him out of work. He joins in on the criticism.

“All those cross-links between the data model elements are supposed to be one-to-many. If I link from element A to element B, I want to have some unique key in B that I can reference. That way, when B changes, we can find the incoming links and keep the data consistent. How are you going to have users maintain those keys?”

Sloth is sheepish now, but still defensive, so he sticks to his proposal. “They can just use whatever they want as keys. Any string would work. They just give a name to element B and then refer to it by name in A.”

“I dunno,” says the DB guy, “it seems to me that we’re going to make it really hard to maintain the consistency of the data model when users are mucking with the keys. If they want to give a name to a model element, that’s fine, but I don’t want to use their name as our internal key. What if they use the same name for several elements? Would you allow that? How would we figure out which is which?”

Sloth shoots himself in the foot: “I guess the users will just have to resolve those problems themselves. They can pick non-colliding names, maybe based on the directory the data file is in? And I see your point, there’s no good way for us to ensure the integrity of the data anymore, but that doesn’t seem to be a requirement. Maybe when we export the data to the upstream system, we can flag all the problems and make the user fix them then.”

“Oh god, you’re retarded.” says the rest of the team, nearly in unison. “We’re not doing it that way,” says the tech lead.
❞


So in our allegory, Sloth gets stuck on his idea of having a cool DSL for encoding the data model, and by the end of the meeting, he’s killed the referential integrity of the system. What would you have done differently in this design? I know one thing I would do, even if we want to use the VCS to store some serialized files like Sloth suggests, is to show the users a more conceptual view of the data. It’s crazy in a system of this complexity to have no distinction between the form of the long-term persisted data and the presentation we show the user. Right?

Well, in my cynical way, my allegory is really about source code and IDE’s. The poor users editing serialized data files in a DSL are you and me. If you look at it in a new light, like in this story, it sounds pretty crazy.

This whole notion came to me in a conversation with Jesse Wilson. The idea of ditching the serialized text file as the encoding of a program is entirely his. At first, I was really skeptical. It doesn’t seem to buy us much, and comes at a great cost. We have to ditch so many tools that rely on text files and directories. We also have to use an IDE that understands whatever alternative, better way we can come up with to encode a program.

But, the more I think about this, and the more I work on the grammar for Noop, and the more I watch myself and my co-workers massage the ascii text in a source file to align parameters, keep the imports tidy, and so on, the more it makes sense. And we could eliminate a lot of complexity from all our tools. Refactorings become easier to write, and our IDE doesn’t need to do incremental parsing of our text to keep its AST in sync. Even better, our IDE can become a lot smarter about how the program is presented to us, the users. I’m going to mock up some pages to show what I mean, sometime soon.

The biggest drawback is that you can’t edit this code on the command line anymore. You need to use an IDE. If you want to grep or sed your way through the files, you’re probably out of luck, unless you want to do it on the persisted form of the data, which might not be very human-readable. Looking towards the future, though, I see a lot of the compilation and testing steps moving to cloud-hosted farms, letting us increase the size of our software and its transitive dependencies. I see a need for tooling to get more sophisticated around helping us visualize and manage the interactions in our code. And I think we could spend less time thinking about how to format this text file full of code, organize our imports, remember that each file needs its own copyright, worry about whether the number of methods in this class makes the file too long, whether we want to make this new thing a public static inner class just to avoid making extra little files on the filesystem, and so on.

Another big win: we don’t have to have compile errors. If we manipulate the AST directly as we code, then the referential integrity of our program can be maintained with each transformation we apply to the source. We would never write software that forced the user to reconcile a bunch of data integrity problems at the end of a long series of operations, like we do with the compiler today.

In my last post, help, my IDE is full of compile errors, I wrote about the problem of keeping our IDE’s fully configured correctly for our project, and sharing their understanding of the project metadata with the build system. Well, if we are going to change the IDE in a major way by using a different data representation for our code, then we may as well have a server between the IDE and the VCS persistence layer. Now the server can maintain the correct state of the metadata for us, and keep it consistent across the team, and across my several machines.

I’m having trouble keeping this urge from taking root in Noop, my programming language. To be fair to Noop, this isn’t a language problem per sé. Java could be made to work in this system – mostly. We want a unique key for an identifier other than a user-defined symbol, so we’d have to do some translation of the AST between parsing the Java files and showing to the user. Also, to get finer-grained versioning, we’d have to break up methods into their own files, so the actual .java source file would be produced from a number of files. It just seems like a new language gives us a new start, and would save me time and effort trying to find a pretty way to serialize all the metadata like documentation into the source files with a complex grammar to maintain.

We could also re-write the VCS so we don’t need to serialize anything to files. But that seems pretty crazy, right?

Tags: IDE

Help! My IDE is full of compile errors!

I was working on some code just now, in a project I don’t normally work on. It turns out I need to make some changes there, and once I’m done, I won’t need to work on that project anymore. This is a common enough thing for engineers, I’m guessing.

So I loaded up the code in my IDE so I could start making sense of it. I followed some instructions provided by the project team, as far as what to check out from version control, and then I used the plugin in my IDE that deals with the build system, hoping to setup my project correctly. And by correctly, there are a bunch of things I need to be happy and productive:

• All the right code is displayed, including all the source roots and modules
• All the needed dependencies are added as libraries, so the code compiles
• If any special development kit is needed, like the right JVM, it’s added
• Whatever plugins I’ve used when I worked on this project in the past should be installed and setup
• If there are multiple modules/sub-projects, they are also added correctly, and depend on each other as source, not via their compiled jars
• Source attached to all the binary dependencies, even third-party ones
• All the transitive dependencies are downloaded to my machine
• The right preferences are available, such as the Copyright block that should be pasted into new files, code style, import order, etc.
• If there are any special arguments needed to launch the program or the tests, those are added in my Run Configurations.
• Whenever I change any of these things, other people on my team have their IDE , and when they make changes, my IDE setup is updated

When you really get serious about your IDE being set up correctly, the state of your IDE becomes valuable. You really don’t want your setup to get borked, or to start over on a new machine, because it takes you a while to get through all these items at the various point they show up. Some of them never get set up right, like all the code formatting preferences, and we just slog through it with a partially-configured IDE.

This valuable state isn’t committed to the version control system, at least typically, because much of the information is duplicated from the files used by the build system to compile and test the program. That part of the IDE setup should be created or synced by some tool associated with our build system. But the rest of the settings are up to us; the build system doesn’t care about them, since it has a read-only view of the sources.

Why is this a big deal? When you’re working on a very large, modular, corporate codebase, you only have a nice, happy IDE configuration for the part of the project you work on regularly. You’re willing to put in that ramp-up effort on one machine, for that one module or subset of the code, and only for one version of the IDE, etc. You hope it’s only once. But what if you pair with other members of the team, their setup might suck, or if you’re trying out a new version of the IDE, or have the project setup on a home computer and a laptop as well as your work machine, or (and this is the real stretch), if you’re doing a code review and want to browse the proposed changes in your IDE, maybe experimenting with some suggested improvements and running tests? These are all times when you have to work in an IDE-not-happy state.

As much as I appreciate that we all have different preferences for our IDE, and that we take our position very strongly, I think that’s the major contributor to this problem. If we had a true standard for project metadata, which allowed for the entire spectrum of metadata I bulleted earlier, and worked with both the build system and the IDE, then all the tools would work fine. But not all the tools cover the whole feature set – if the copyright on our project changes, IDEA 8+ users want their metadata to reflect that, but no one else has a box in the IDE to update. And we don’t want to go monkey with that project metadata by hand, since in most IDE’s and build tools, it’s not very friendly. (Unless you’ve got a nice XPath/XQuery library in your head)

What’s the answer to this problem? Sadly, we have two impossible tasks: pick a build system and an IDE that we’ll make everyone use, and get that build system and that IDE to share their metadata, either by using the same location for it, or at least syncing everything correctly. I personally believe that if we really solved this, any coder who knows how to use an IDE well would appreciate the convenience of complete and correct metadata so much that it would be a ‘killer feature’, and they’d drop their IDE. Crazy talk, yes. But I think there’s more we can fix once we go down this path, which I’ll write about next time.

Tags: IDE

A test-driven modern language

Here’s another in my recent series of posts about ways I think the Java language could support modern coding practices.

An obvious practice that we use extensively today is unit testing and test-driven development, and again, Java and other languages don’t provide built-in support. Instead, we have a few libraries to create test code, execute it, and provide mock dependencies, and then some standards for naming and directory layout that help us organize the test code and correlate it with production code.

What sorts of facilities would our dream language have for testing?

In Java, tests have a lot of repetitive code, because they are just classes. All test methods are in the same form, namely that they are non-static, take no arguments, marked as a test via an annotation or naming convention, throw all checked exceptions, and return void. We should probably have a test keyword that starts a block, and this should act like a test.

Just like the keywords extends and implements, we should have a way to associate two objects with a new relationship: tests. This way, we can easily see for a given class, what classes test it, and IDE’s can consistently navigate between a class and its test(s). It also would allow some conveniences. Instead of having to mark methods package-private so they are visible to the unit test, we could make private methods visible to tests, like an implicit C++ “friend” relationship.

So, with our two new keywords we have an example like this:

class Foo {
  private int helperMethod() {
    //stuff
  }
}
class FooTest tests Foo {
  test helper {
    int i = new Foo().helperMethod();
    assertThat(i, equals(10));
  }
}

Now we have created a test class and some tests. Notice the assertion in that test – where do the assertThat() and equals() come from? Instead of importing some utility classes, maybe the tests keyword could also cause my FooTest class to extend from a base Test class rather than from Object by default, so these could be implemented there. Or if the language had mixins, the test methods could come from a mixin and not require polymorphism (we don’t care that our test class may be cast to a Test type).

The example also uses a Hamcrest-style expectation, to make the assertion more fluent.

The same thing could also be done if tests could be written directly in the class they test, and the compiler will have to ignore it when not in testing mode. In that case, the mixin would be automatically added to classes that contain tests, so that the assertions are available:

class Foo {
  // implicit mixin Asserts
  int calculateDay(Date date) {
    ...
  }
  test calculateDay {
    // using the enclosing instance, requires that it have a default constructor
    int day = calculateDay(new Date()); 
    assertThat(day, equals(31));
 
    // probably more realistic
    int otherDay = new Foo().calculateDay(new Date());
    assertThat(day, equals(31));
  }
}

What more can we do to make writing tests the most convenient and fastest way to code? For one, we could have a built-in test runner. Having marked our tests with a new syntax, we can find all the tests and execute them using an equivalent program to the compiler. In that test running mode, we can relax the security model of the runtime, avoiding some common problems with testing sensitive code.

Once we have our test runner, we can lower the bar to getting good testing infrastructure in a project. One major pain in the butt is instrumenting the code to get the line-level test coverage. Because libraries like JUnit execute code that’s been compiled by the normal compiler, we have to do something funny like twiddle the bytecode after compilation, or use a custom classloader to do it on the fly. If the language understood testing, then it could also always provide coverage data when the tests are executed. It would be easier for IDE’s to show how much of your code is executed in the tests, as well.

Finally, there is the ever-annoying issue of setting up dependencies. The language needs access to different libraries at test-time, and we also want to enforce that production code may not have dependencies on test code. With language-level support for testing, there can be compile-time checks that test code is not used from production code, and we can have a second path of libraries passed only to the test runner.

Does this seem like a good idea, or does the language step over the line here and take over the job of a framework? Should we encourage testing private methods this way, or does it violate encapsulation principles?

Tags: nil, testing

Simpler dependency injection

In my last post, I wrote about how Java’s final modifier could be replaced with a mutable modifier, helping us to follow modern coding practices.

Another feature Java and other languages lack is a built-in understanding of dependency injection. I’ve been using Spring and Guice for a few years now, and it’s so clear to me that dependency injection is absolutely essential to writing a complex application. Even more than separating code into classes, we must avoid mixing the wiring with the logic. It makes code much more testable – if you look at the Testability explorer metric, it is partly based on the expense you incur in testing when a dependency cannot be injected – you are bound to test the object with that particular collaborator.

In Java, a typical incantation of constructor injection looks like this:

class Foo {
  private final Dependency1 d1;
  private final Dependency2 d2;
  public Foo(Dependency1 d1, Dependency2 d2) {
    this.d1 = d1;
    this.d2 = d2;
  }
}

The only information we are providing here is that to construct this class, we’ll need access to two dependencies, and we’ve given them names. This is how most classes should start, and the list of dependencies can be long. Why not treat this just like arguments to a method? We just provide the types and names of the arguments, and then they become new variables in method scope.

What we need to improve this is Properties. Just ask Java Posse co-host Joe Nuxoll – Java needs properties. If you don’t know, a property is just like a public member variable, except that you can write a set or get method for it later if you like. It replaces the silly Javabeans spec – to client code, it looks just like a public member, you just use object.property = 1 to set and object.property to get. I like how they are implemented in ActionScript, with set and get keywords to override the assignment/dereference:

public var one:int;
private var _two:int;
public function get two():int {
  return _two + 1;
}
public function set two(i:int):void {
  _two = i - 1;
}

Now we can dream up an example:

class Foo(Dependency1 d1, Dependency2 d2) {
  void doThing() {
    d1.print();
    d2 = null; // not allowed - the property is read-only
}
Foo foo = new Foo(new Dependency1(), new Dependency2());
foo.d1; // also not allowed - the properties are private

This would create private read-only properties d1 and d2, and a default constructor that requires both to be passed in. It’s important that the values are final – we want to reduce the mutable state at runtime, and assigning the properties during construction allows this.

Ok, now setter injection: if the dependency is optional, meaning we could have a Foo in a valid state without it, then it might be appropriate to inject the dependency with a setter instead. (For some reason, setter injection seems to be the preferred way in Spring – which is wrong.) We could allow a convenience for setter injection as well, simply by adding a default value for that dependency in our “class arguments”:

class Foo(Dependency1 d1, Dependency2 d2 = null) {
}
Foo foo = new Foo(new Dependency1());
foo.d2 = new Dependency2();

Now the default constructor takes only one argument, of type Dependency1, which populates the readonly property d1, and the read/write public property d2 starts off as null. Because it’s a property, outside code can set it. The only aspect that’s missing here is if you want to have additional constructors – but with no additional syntax, you could just write those constructors out in the class body.

The more I have thought about this, the more obvious it seems that injected dependencies of a class are analogous to method parameters, at one scope higher. Why not make the syntax the same, so that this good practice is the easy way? I’d love to hear whether you think this is a good idea.

Tags: injection, nil

Java’s final modifier is backwards

In Java, there’s a modifier that can be applied to a variable, and it’s called final. Final can also be applied to classes and methods, to prevent overriding, which is kind of overloading the word. I don’t have a problem with that kind of final, except that I don’t agree that it’s good to use it liberally where you don’t design for inheritance up-front. This post is about final variables and immutable objects.

Good practice dictates that variables should always be marked final, except where the reference needs to be changed. This is true for class members, which should be assigned in instance initialization or a constructor, and can therefore be marked final. It’s true for method parameters, where it prevents you mistakenly thinking that Java is pass-by-value. And of course block-scoped variables should be final. All of this is good because it reduces mutable state at runtime, making your app less complex, more testable, and expresses your intent in compiler-enforceable constraints.

class Bear {
  final DateFormat df = new SimpleDateFormat("MMddyy");
  final SalmonService ss;
  Bear() {
    ss = new HuntSalmonService();
  }
  void eat(final Stream s) {
    final Integer hungerLevel = 10;
    ss.takeFish(s, hungerLevel);
  }
}

The problem is, I only know one coder who is religious about declaring everything possible as final. Which sucks: if a variable is NOT marked final, this tells you nothing. Most likely, the coder didn’t write it because they were in a hurry or didn’t think it was needed, but they never modify the reference. The fact that Java makes you write this modifier everywhere is backwards, because you are rarely forced to use it (for use in anonymous inner classes). It would make more sense to use a modifier to note the exception rather than the rule, so we should have a mutable modifier instead. Now if you see a mutable List foo, you can bet the reference is really changed to another List later, since the developer bothered to mark it as mutable.

In Scala, you are forced to think about this, by marking your identifiers as a var or a val. Val’s are final, var’s are not. That’s good, Java should do that too. Now our Bear example is:

class Bear {
  DateFormat df = new SimpleDateFormat("MMddyy"); // non-mutable
  SalmonService ss;
  Bear() {
    // Without this, compiler complains that ss must be marked mutable
    ss = new HuntSalmonService();
  }
  void eat(Stream s) {
    // re-assigning s here would be a compiler error
    mutable Integer hungerLevel;
    hungerLevel = 10;
    ss.takeFish(s, hungerLevel);
  }
}

So far, this would be an easy change to the Java language, or something a new language could model.

There’s a second problem, however. I love to ask this in interviews: if something is marked final, can it be changed? It’s intentionally vague, and of course the answer is that the reference cannot change, but the value can. Immutability is important for the referenced object even more than for the reference, since immutable objects are thread-safe, can cache expensive operations, may be pooled, and it’s easier to reason about their state. They also have the same benefits as final variables – less mutable state at runtime is a Good Thing. Is there anything to be done at the language level to support the maxim, “Favor Immutability”?

Let’s imagine what it could look like. We’ll add the mutable modifier to a class that is mutable:

mutable class MyList<E> implements java.util.List<E> {
  List delegate = new ArrayList<E>();
  public void add(E item) {
    delegate.add(item);
  }
  public E get(int index) {
    return delegate.get(index);
  }
  // More implementation for List methods ...
}
 
class YourList<E> implements java.util.List<E> {
  // We don't even want a mutator, but have to satisfy our interface
  void add(E item) {
    throw new UnsupportedOperationException();
  }
  public E get(int index) {
    return delegate.get(index);
  }
  // More implementation for List methods ...
}
//reference and value are both mutable
mutable List<Object> foo = new MyList<Object>();
foo = new YourList<Object>(); // now the value is immutable

This is no different from what Java does already: you have two classes, one that is mutable, and one that isn’t, or a method like Collections.unmodifiableList() that returns a wrapper around a list with exceptions thrown by the mutator methods – in the latter case, you don’t find out until runtime that mutation was attempted. Just like with the final modifier, the status quo is that everything is mutable, and thoughtful programmers sometimes use a utility method to produce immutable instances or have to write a second copy of their classes that lacks or hides the mutator methods. Note that there are lots of classes named “Immutable*” out there – maybe that’s backwards too. Again, we’d like the language to give us immutable objects unless we specifically ask for a mutable instance.

So we’ll just have the one MyList class, and we’ll need to mark the methods that are mutators. Those methods are only available, at compile time, if a mutable instance of the object was constructed. Since the mutable keyword is already in use for the referencing variable, we’ll need some other way to request mutable instances, maybe a mutable version of the new() operator. We can also mark methods as returning mutable instances, since we have made mutability part of our type system, and we don’t want callers to be able to cast an immutable return value into a mutable instance.

class MyList<E> implements java.util.List<E> {
  List delegate = new ArrayList<E>();
  mutator void add(E item) {
    delegate.add(item);
  }
  public E get(int index) {
    return delegate.get(index);
  }
  // Return a mutable copy:
  public mutable MyList copy() {
    // call the copy constructor, 
    // using the mutable() keyword instead of new()
    return mutable MyList(this);
  }
  // More implementation for List methods ...
}
mutable List<Object> foo = new MyList<Object>();
// Compile-time error: This instance of MyList is immutable
foo.add("Not allowed");
 
foo.copy().add("Can do this");
 
foo = mutable MyList<Object>();
foo.add("Good");

Is this possible? I’m still pondering the implications. Here are a few:

• Can a non-mutator method call a mutator method? Probably that should be compile-time error.
• Should support reflection on whether an instance is mutable.
• Can the compiler figure out what are mutators without marking the method? Right now, if we forget to mark a method, it violates the immutability of our object. Or it could be reversed: a modifier could be required for the methods that are legal on immutable instances.
• Maybe if a newly created object is assigned to a mutable variable, it should be mutable? Then we could avoid this new brother of the new() operator.

I’d love to hear your comments.

Tags: final, immutability, nil

Use Hibernate narrowly!

Hibernate is a good library. Sure — I first loved it too much, as many of us have. At first, it seems to be the magical solution to the whole persistence layer, probably the most complicated and potentially troubling part of an application.

In the first stage of falling out of love with Hibernate, you discover that there are some aspects of persistence where you need to understand what you’re doing, and Hibernate doesn’t necessarily make it easier. My big awakening was optimistic locking: if you have the same entity in front of two users, how do you avoid the second person’s “save” action overwriting the first? In Hibernate, you add versioning to your entities, so that a new database column contains some unique, incrementing identifier that needs to match the property of the object that’s to be saved. This caused some big headaches with my client at the time, who had a screwy database schema where a created_date and modified_date on each row of each table were not both filled in for a new entry created in the Oracle Forms app, so neither column could be the versioned identifier for Hibernate.

The second stage of falling out of love is where you want to use Hibernate in a non-CRUD scenario – you are not showing individual records to a user for them to be edited. Data import, for example, is one of these things where Hibernate may do things you’d rather it didn’t. Even after disabling the second-level cache, Hibernate was still holding onto some reference to each entity. It wants to manage the objects it persists, and if you’re saving thousands of objects to the database,you’ll probably blow the heap. No doubt there was a fix for this — but even after using a StatelessSession (which helpfully doesn’t implement Session) I still had problems.

The moral is, keep in mind that JDBC, with a good library providing boilerplate templates like Spring-JDBC, is still the right tool for many jobs. I’m often guilty of wanting to apply the new tool (hammer) to all problems (screws). Don’t be that every-problem-looks-like-a-nail guy. It’s immature.

Handy tip for layouts in Freemarker

If you’re writing a Java webapp and using Freemarker as your templating engine, hooray! Skip to the next section. Maybe this tip will come in handy.

If you’re not using Freemarker, you should really consider it. JSP’s are ridiculous – why are you compiling all your markup into Java code? Do you like it when you make a small typo in your JSP page, and get an error from the JSP engine referencing the Java code it created to render your page?? Gross, let’s go read through the generated code to infer what the typo was! Do you like how the JSP pre-compilation step is annoying to set up, but if you don’t, you have the dreaded first request to worry about? JSP’s might be a little faster than interpreting markup on each request, but with smart caching, and the ease of adding more web server boxes, that’s a dumb reason. And wow, let’s give people the ability to write arbitrary controller code in the view, and then trust them to do things the right way instead.

Oh, and if you like Velocity, that’s cool I guess. Freemarker is nice because it’s a real stickler for correctness – a null value isn’t just rendered as a blank string, because it’s not (insert scary life-support app example). The model is the model, you can render it, and that’s all. Just what you should be doing.

The layout tip

If you’ve played with (or gotten paid to work in) Ruby on Rails, you’re familiar with the “layout” concept. It’s just a simple template for a page, and instead of using “includes” where the template has to know what’s going to be inserted, it just “yields” to the page content. It’s the same philosophy as Struts Tiles – the dependencies between your markup files should go from the bottom up – the content needs a place to be inserted, and the template doesn’t care what it is.

This layout method gives you a similarly easy and readable result in Freemarker.

First, let’s create a page, with some wishful thinking for how we think the layouts should work:

<@layout.basic>
 
<a href="/customer/accounts">Browse accounts</a>
 
</@layout.basic>

That’s saying, I’d like to use a layout named “basic” and in the place that layout puts the page content, I’d like to have my link to Browse Accounts show up there. The page is as short as can be, and readable.

Now, let’s make it work. It seems I’m calling a macro “basic” and I expect it to be in the “layout” namespace. I haven’t used an import macro on my page, so the code for the layouts will be auto_imported into my pages, using the “auto_import” setting in Freemarker’s Configuration object. I set it to “/layouts/layouts.ftl as layout” so that the template /layouts/layouts.ftl gets imported to all my pages with the layout namespace. (It has to be that way because Freemarker can only associate one template with each namespace.) If you’re using Spring to configure Freemarker, you’d create a props element containing an “auto_import” prop and pass it to the freemarkerSettings property of the FreeMarkerConfigurer bean.

Now I create the layouts.ftl template. This could contain all my layouts, but it would look strange for several of them to live in one file, so instead my layouts.ftl looks like this:

<#include "admin.ftl" />
<#include "basic.ftl" />

Clearly this is where you add additional layouts as you need them.

Finally, let’s look at the contents of basic.ftl:

<#macro basic>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  </head>
  <body>
    <div id="header">
      <p>
        Cows make moo sounds
      </p>
    </div>
    <hr/>
    <div id="content">
      <#nested>
    </div>
  </body>
</html>
</#macro>

This creates a macro named “basic” and uses the nested macro to include whatever appeared in the body of our caller’s <@layout.basic> tag.

Neato!

Choices, choices

I’ve been thinking about the choices programmers have to make eventually. I guess this assumes that you’re somewhat talented, ambitious, and thoughtful, because I’m sure a lot of coders can just sit happily at a desk and not ask themselves this. Also, it’s just one way to divvy up the options, there are probably better ways.

I don’t feel like writing pros and cons, so I’ll just give cons and you can use your imagination.

Product company or Consulting?

Product company

You will have to work on the same thing for a long time. At first, when the product is under development, there are a lot of cool things to try and big decisions to make. But then it gets boring, you have production support to do, and some of those decisions turn out to have been wrong and can’t be easily corrected. The bottom line for the business at this point depends more on sales of the existing products, so the emphasis on cool technologies is diminished.

Because you’re supporting existing code, it’s hard to find a chance to learn new stuff. Maybe you can deceive your manager into using the stuff you want to learn about, under the guise that the system needs to be re-written and this technology will save a lot of money or fix a lot of problems.

Consulting

You’ll have to work with “non-ideal” client’s employees. If the client hired a lot of really smart developers, they wouldn’t need the consultants. Or maybe the really smart developers know to stay away from working there, so they need to pay the people who are willing to work anywhere as long as it’s only for a year or less at a time. This makes it harder to be challenged by your peers to improve – if you’re spending most of your time working with people who are not smart, you tend to go down to their level instead.

You’re pretty much guaranteed to work on junky code. Again, if they had to hire consultants, they need your help to solve the challenging problems. It’s probably not an interesting new business idea being implemented, more likely it’s a self-created problem with their monster spaghetti codebase and vendor-provided toolset.

You have to give up some of your ideals about the quality tools/platforms you prefer. If your company gets a .NET contract, and your client insists you use Windows, then guess what? You’re going to use a closed platform with second-rate versions of your favorite tools (if there is an ‘N*’ of your tool) and you’re not going to use your Mac even though it’s your more productive environment without a pile of bugs. And if the client is using the Oracle J2EE app server, then you’re going to be decompiling it. This is a tough one, since a good developer stays agnostic about the tools and platform – but sometimes you have that ideal for a reason.

Many people don’t want you there. Sometimes this includes your client-side supervisor, who was told to bring in consultants by their boss but thinks they could handle it fine on their own. If you produce a report that supports their pre-conceived ideas, then maybe they’ll be excited about having you there, but more likely you’ll be creating work for them and telling them things they don’t want to hear. And of course the client’s employees are mistrustful and think you’ll make them look bad. And they probably value job security more than you do.

Keep coding or move to management?

Keep coding

You may get stuck as a “line programmer”. This is like modern factory work. Your job is outsourcable or offshorable because there’s a heavy reliance on “requirements” as an excuse to do waterfall, so the programmer doesn’t have an interactive role and anyone can do it. This happens because many people think programming is typing and will never understand the art of it.

There are few places where you can move up the pay grades as a programmer. The Java Posse did a listener feedback question on this once, and said if your company doesn’t promote really good developers as highly as managers, then you should move to another company. Problem is, Sun, Apple, and Google are mostly doing this work in the Bay area so that leaves a lot of people out. I’m still hoping there are more places like this.

Also, you may not get a chance to be a leader or promote your ideas. Developers don’t have much decision-making power.

Management

You will probably become a bullshitter. Since you are disconnected from coding, you won’t know exactly how the software behaves in a corner case, or how difficult a feature would be. So you make stuff up and make educated guesses that become random guesses over time.

You’ll have to give up the dream of being the next Gavin King or Craig McClanahan or other coder hero. Instead, you will be driven to improve non-programming skills like corporate finance and negotiating.

The business environment is cutthroat, which doesn’t match well with hacker ethics.

Worst, you won’t do what you love, you will write proposals and performance reviews and project plans and push tickets and schedule releases instead.

Use left-side or right-side of your brain?

Right-brained – the Artist

It’s hard to find a paying gig where you can invent the next great thing (or be a part of it), so you have to do one-off contracts and either manage a lot of business stuff yourself, or be a starving artist type. Until you get your job at Apple, where I think most of the programmers are the artist type.

It’s not clear how to pursue an artist/hacker ethic. You can work on your brilliant idea at home, but where are the interviews for this sort of work? Perhaps there are not many openings and a lot of applicants, like aspiring to get in the NBA.

You have to gain true respect of your peers, for example, become open source committer on a well-known project. The artist rejects unearned corporate rank and title, especially when bestowed by someone who doesn’t recognize quality.

Left-brained – the Analyst

You won’t “save the world” – you will create predictable products that mimic others.

There may be pressure to create average-quality products, since there is no business need to exceed customer expectations, especially if it makes development more expensive.

Use a $%*& SocketAppender

Pretty much everyone uses Log4J, hopefully through a facade layer like Commons-logging or JDK1.4 Logging. However, almost no one I know uses the SocketAppender.

Everyone has to establish a balance between using the debugger and writing log() statements. It’s annoying to leave behind logging that was relevant only in the context of fixing a bug, but also bad to have no data on production issues because your debugger was your only source of diagnostics while you coded.

So, assuming you have some log statements, you ought to be able to watch them on one screen (you have several, right?) while you navigate through your app on the other. And, when there’s a stack trace, the log entry should still only occupy one line on your log viewer, and you can summon a nicely formatted stack trace for that line when you like.

Too many developers are ignoring logs (how many times do I have to answer your question with “did you check the log?”) and about as many are eye-parsing through log files. Use Chainsaw or Lumbermill already!

It’s pretty easy to do:

<appender name="lumbermill" class="org.apache.log4j.net.SocketAppender">
  <param name="RemoteHost" value="localhost"/>
  <param name="Port" value="4445"/>
</appender>

Also, why is no one maintaining the log viewers? I tried to take over Lumbermill, which hasn’t been active for 1.5 years but I thought I saw Robert Bushman claims he’s still working on it.