Marick’s Law: When it comes to code, it never pays to rush

Thou shall not access network resources from unit test

Currently I am working on a tool called Verde, this is an automated characterization testing tool targeted for all Java legacy applications out in the wild.
One of the key feature of this tool is it can generate characterization tests with mocks meaning if we want we can generate Junit test cases by mocking database access or network access. Yeah it is cool.

As part of our integration test suite of Verde we have few end-to-end tests that generate test cases with mocks from sample application and run them to verify that everything is working, as they should be.

Recently one of our developer noticed that generated tests are no longer generating correct mock expectations. This must be because of some changes we made in our code but the crazy part is our integration tests never failed. In spite of wrong expectations our tests are still running fine because without mocks our generated tests are hitting the actual network resource and asserting the result received over the network. So from outside everything looked fine and worked fine too. Now this is an interesting problem, we cannot disable network connection because some tests are expected to use external resources. What do we do?

SecurityManager is for reuse.

Without going into details of how Security Manager works in Java, java.security.SecurityManager defines quite a few check methods for various operations like file read/write, print job, network access etc. The behavior of these check methods is very simple, if allowed nothing happens but when denied these methods are suppose to throw SecurityException.
So to tackle this problem we created a custom security manager to block network access. First we overridden all the check methods with do nothing implementation. This means that you are allowing all the access or granting all the permissions. Then we did something like following.

…

@Override
public void checkConnect(String host, int port) {
throw new SecurityException("Not allowed to access network connection, make sure all the dependencies are mocked out");
}

@Override
public void checkConnect(String host, int port, Object context) {
throw new SecurityException("Not allowed to access network connection, make sure all the dependencies are mocked out");
}

…

We are throwing SecurityException from couple of check methods related to network access. What it means is now when test will try to access network it will get security exception, and the test will fail. That’s it. Maybe oneday you might find this trick useful.

Oh don't forget to specify your custom security manager using –Djava.security.manager when running the tests.

Offline website browsing

There is nothing like free books, especially when it is really a good one. Recently I have started playing with Haskell; a pure functional programming language and the best book I found so far is http://book.realworldhaskell.org/ (the printed version is not out yet). Unfortunately this book is only available online and I really wanted to read it on my way to India. While I was feeling disappointed about the lack of internet accessibility on airplanes I found something that is really cool, HTTrack
This tool allows you to archive the whole website for offline browsing. If you are unhappy because you didn’t find a Mac version, don’t be, there is a macport available. Now I don't have to watch those boring inflight movies anymore, I can now play with something else.

Another programming language for JVM

Things are really going crazy in programming language world these days, especially for JVM. Yesterday I noticed the new loke programming language from Ola bini and today I found clojure.
loke is a prototype based object oriented programming language. I personally think prototype based languages are much more fun to play with compare to class based object oriented languages, the reason I love JavaScript so much.
On the other hand we have clojure which is a functional programming language with lisp dialect. I haven't looked into it yet but I will definitely play with it very soon. Infact there is already a beta book available for it.

I am not sure what tomorrow has in store for me?

Trust in team environment

The following paragraph kind of sums it up when people ask what trust means for teams?

I believe you are competent to do the work

I believe that if you have an issue with me, you’ll bring it up directly
with me, not talk behind my back.

I believe will follow through on commitments—or let me know when you need
to renegotiate.

I believe you have good intentions towards me and the team.

I give up...mysql

Its over with mysql. I hope that I don't see it again. There are things that you love about database and installing them is not one of them, especially when it comes to MySql. No no I am not trying anything fancy, all I want is to get Mysql up and running on my new Mac book pro.

The installation process goes through fine but when time comes to login, mysql comes back with aceess denied for 'root@localhost.com'. According to the documentation, root user comes with no password...hmmmm, never mind. Lets move on.

My first reaction was , maybe its using some old my data folder that I migrated from my old mac book pro.

So I tried running /bin/mysqld_safe --init-file=reset-root-password-file to reset my root password. The init file contains an update sql statement to update mysql.user table with a new password for root user. To my surprise the situation did not improve at all.

Next I tried starting the mysql deamon server with -skip-grant-tables option and it worked. No don't start celebrating yet, when I said worked I mean now I was able to connect to the server using my mysql client. And when I looked up the user table I discovered that I don't have any users. Why ? I really don't know. At this point I have already spent 2 hours figuring out the problem but not ready to give up yet. I looked it up on google and as usual I am not alone, some mysql installation doesn't run the post installation script to create users and grant permissions.

All mysql installation comes with a mysql_install_db script under mysql/scripts folder. Apparently for some installation you have to run it manually. I am not sure why?.

Anyways, finally I thought I got it. So I went ahead and ran the mysql_install_db, it fails. This time because of 'duplicate entry for localhost-'. Apprently this happens when your hostname is set to localhost. Maybe this it, so as recommened I changed my hostname to my name and hooray the script ran successfully. Now I am ready to build the rails app I was planning (the only reason I want to use mysql in first place is because my pair has that setup) for in the first place. But no...now my mysql server doesn't start up and it doesn't even show any error message.

I think 3 hours a lot of time to setup a database and I cannot take it anymore

What makes design...simple

Last weekend in the “Simple Design and Testing Conference” we had a discussion on “What makes Design Simple” and this is a summary of the discussion we had along with my thoughts.

Well before we go further with in our way of defining simple/complicated design we have to step back and first answer why do we design?

We design to be productive and I don’t do it if it is below the DesignPayoffLine. Now if you read What is software design which I highly recommend you will learn that we design all the time, when customer approves a story he/she makes a design decision (this is interesting, some times we evolve to complicated design when stories are not broken properly), when we write tests (we do test driven development) and programming is also a design activity. So I guess we are always designing with or without knowing it.

Being an agile developer, we don’t focus ourselves on BUFD (Big Up Front Design) but rather we believe in evolutionary design, a design that grows as your system grows and there are benefits of doing that which I will not discuss here (read http://martinfowler.com/articles/designDead.html). One of the key aspects of the evolutionary design is to do the “simplest thing that could possibly work” nothing less, nothing more. There are couple of things that needs to be noted here, “simplest” and “possibly”. Simplest or simplicity is your shortest path to the solution i.e. getting your idea from your head to screen as quickly as possible. And why it is possibly? Because we are not sure whether it will work and that’s why we write test to make sure that our idea works.

But the interesting part is we all start simple (I am totally ignoring the fact that there are teams that start with 200 page design document from day one) and it gets complicated over time because it starts doing more work. But if we could restrict ourselves to keep it simple for tomorrows need tomorrow then I guess we can preserve simplicity and this is where refactoring step fits in. Refactoring is our way to simplify things when it gets complicated. Einstein once said, "As simple as possible, but no simpler." And I think that makes sense here.

Wait a minute we haven’t defined simple design yet? I don’t think I can define simple in two lines but lets ask the question differently, “How will we know that our application design is simple?”

Software with simple design will enable team members to quickly respond to change. And as your design deteriorates, so does your ability to respond to change. This is really critical for agile projects (again ignoring rest 70% of teams who don't believe in agile, they really don’t care) because we embrace change and customer satisfaction is important to us. Kent beck calls it “reversibility” property of software. If we can change our decisions quickly it is not a big deal to get them right in the first place and that means less planning upfront (we are wrong most of the times anyways). There are some really good second order benefits to simple design, we get our work done sooner and it is easier to communicate our work with others (I am not sure about you but I still work as part of a team, collective ownership rings any bells?). Above of all it is less stressful to work with simple design.

So how can we keep our design simple? Testability

I absolutely agree with James Bach on " there are no best practices" but rather appropriate practices and I think the appropriate practice to keep your design simple is continue to write tests.

Testability is a characteristic of software application that provides ability to developers to write tests for their software application. In the world of TDD, we derive our design from tests and having that ability is very important. In fact if it is hard to write test, developers will stop writing tests especially new agile developers. It is very easy to get into “code and fix” mode under agile radar. If you are the only manager reading this article please pay attention to your developers when they say it is hard to test, this could be good indication that application design is deteriorating.

While discussing “What makes design simple” in the conference, Brain Marick raised one interesting point, we should strive for habitability, “The characteristic of source code that enable developers feel like home and place their hands on any item without having to think deeply about where it is”( read http://www.wirfs-brock.com/PDFs/DoesBeautifulCodeImply.pdf). We do spend good portion of our time everyday inside the code we are working on and habitability makes working easy. I guess more appropriate answer to the question “How will we know that our application design is simple?” would be habitability. If it is easy to work with your design, it is simple.

If you are still with me then probably some of these makes sense but if you were looking for top 10 practices for simple design sorry to disappoint. The only take way point is “Design for today, not for tomorrow”

If we have to look at the code to explain the program, are we missing a test?

What is the first thing that you know about project before starting it?

due date of the project and reality is nothing goes as you plan in this world.

Using Groovy Script

Groovy Script

Groovy Scripts are different from Groovy classes, in Groovy Scripts you can directly write code without declaring methods or classes (just like your bash or perl scripts). We can use Groovy Scripts from Groovy classes, Java classes or using "java" to run them.

From Java Classes

ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("groovy");
System.out.println("Calling script from Java");
try
{
engine.eval(new FileReader("MyGroovyScript.groovy"));
}
catch(ScriptException ex)
{
System.out.println(ex);
}

Note: Make sure your groovy-engine.jar file is in your classpath

From Other Groovy Scripts

def shell = new GroovyShell()
shell.evaluate(new File('MyGroovyScript.groovy'))

From Command line

We can compile our Groovy Scripts using groovyc compiler and when you do that it will create a single ".class" file based on name of the script with auto generated main(String[] args) method. Once you have a main method we can clearly run using java command, no tricks there. The auto generated main method creates an instance of the class which extends groovy.lang.Script and invokes run method on it (We will come back to this)

From Groovy Classes

def runSomeScript() {

...

run(MyGroovyScript)

...

}

What the heck is this? Are we missing something?

No we are not. In Groovy (like Ruby) class names are constant and in Groovy this constant points to java.lang.Class class of the class created by the groovyc compiler. So now if I have a method like following I would be able to run the script

def run(Class scriptClass) {//You don't have to specify type, I did for clarity

Script script = InvokerHelper.createScript(scriptClass, new Binding()) //Binding is a way to share variables and context info with Script

script.run()

}

View hidden files in Finder

Lately I am having stupid merging issues with .classpath (eclipse) file because of subversion. Anyways that's not what I want to discuss here. In Mac dot files are hidden and you cannot view them in finder by default and my windows friends find that bit annoying while pairing. So I looked up and found that we could use "defaults" command from terminal to change some of these behavior. For example if you want to view hidden files in finder, the following command will enable it

defaults write com.apple.Finder.AppleShowAllFiles YES

To make the above command work you have to restart your finder from your dock though.

Not only that , you have other hidden features that you could enable, especially I liked the one where you could drag widgets from dash board to your desktop.

Even better you have TinkerTool which allows you to play with these hidden defaults from a nice gui...Sweet.

Understanding The Misunderstood, JavaScript Part 1

Many of us work with JavaScript in daily basis but only few actually consider JavaScript as a powerful programming language. But new renaissance of JavaScript frameworks prove that there is something about this language that we haven't learned yet. This mini part series of blog posts will give another shot at JavaScript.

So what went wrong?

JavaScript was developed by Brendan Eich at Netscape as the in-page scripting language and most often used for client-side web development. It is a remarkably expressive dynamic programming language. JavaScript, despite the name, is essentially unrelated to the Java programming language. This language was renamed from LiveScript in a co-marketing deal between Netscape and Sun in exchange for Netscape bundling Sun's Java runtime with their then dominant browser. I think this is the main contributor to the misunderstanding we had about this language.

Most programmers never realized the true power of this language. The reason being most of the people writing JavaScript in early days were not programmers and they lacked the training and discipline to write good programs hence finding good JavaScript materials or examples were hard. Along with those browser incompatibilities, bugging implementation gave an impression that this language is not ready for professional programming. This is simply not true anymore, in many ways we can say that suddenly the bar was put much higher in front of us as we started to see some smart JavaScript programs and applications. And as the web is becoming target platform for Web 2.0 applications, scripting languages and other dynamic languages are currently experiencing a renaissance. JavaScript is becoming the only scripting language of the web and its clearly becoming important to understand this language, its paradigms, and its semantics.

Re-introducing JavaScript

JavaScript is prototype based object oriented dynamic language descendant from Self. Quite a mouthful! right? Lets explore.

If you are familiar with prototype design pattern, you know that in this creational pattern objects are created from a prototypical instance. JavaScript uses the same concept to create new instances, by cloning existing objects that serves as prototype (more about this later). This is quite different from class based languages in which objects comes in two distinctive taste, classes as collection of behavior and instances as state holder.
In JavaScript, objects hold both the behavior and state. This language semantic enables the programmer to focus on behavior without worrying about the abstractions and relationships upfront. Here is nice comparison of prototype-based language with class-based language from wikipedia

Traditional class-based OO languages are based on a deep-rooted duality:
1. Classes define the basic qualities and behaviors of objects.
2. Object instances are particular manifestations of a class.

For example, suppose objects of the Vehicle class have a name and the ability to perform various actions, such as drive to work and deliver construction materials. Porsche 911 is a particular object (in-stance) of the class Vehicle, with the name "Porsche 911". In theory one can then send a message to Porsche 911, telling it to deliver construction materials.

This example shows one of the problems with this approach: Porsches are not able to carry and deliver construction materials (in any meaningful sense), but this is a capability that Vehicles are modeled to have. One way to create a more useful model is to use subclassing to create specializations of Vehicle; for example Sports Car and Flatbed Truck. Only objects of the class Flatbed Truck need provide a mechanism to deliver construction materials; sports cars, which are ill suited to that sort of work, need only drive fast.

This issue is one of the motivating factors behind prototypes. Unless one can predict with certainty what qualities a set of objects and classes will have in the distant future, one cannot design a class hierarchy properly. All too often the program would eventually need added behaviors, and sections of the system would need to be re-designed (or refactored) to break out the objects in a different way.[citation needed]

Experience with early OO languages like Smalltalk showed that this sort of issue came up again and again. Systems would tend to grow to a point and then become very rigid, as the basic classes deep below the programmer's code grew to be simply "wrong". Without some way to easily change the original class, serious problems could arise.[citation needed]

Dynamic languages such as Smalltalk allowed for this sort of change via well-known methods in the classes; by changing the class, the objects based on it would change their behavior. But in other languages like C++ no such ability exists, and making such changes can actually break other code, a problem known as the fragile base class problem. In general, such changes had to be done very carefully, as other objects based on the same class might be expecting this "wrong" behavior: "wrong" is often dependent on the context.

In Self, and other prototype-based languages, the duality between classes and object instances is eliminated.

Instead of having an "instance" of an object that is based on some "class", in Self one makes a copy of an existing object, and changes it. So Porsche 911 would be created by making a copy of an existing "Vehicle" object, and then adding the drive very fast method. Basic objects that are used primarily to make copies are known as prototypes. This technique is claimed to greatly simplify dynamism. If an existing object (or set of objects) proves to be an inadequate model, a programmer may simply create a modified object with the correct behavior, and use that instead. Code which uses the existing objects is not changed.

Like most programming languages JavaScript has a type system but its dynamically determined by the runtime environment, its types are associated with value not with variables.

Some examples would be handy right about now

Since essentially everything in JavaScript is an object it would be nice to have some familiarity to it.

var myJSONObject = {"ircEvent": "PRIVMSG", "method": "newURI", "regex": "^http://.*"};

Here in the above JSON example, object is created using object literal syntax, which is comma-separated list of property/value pairs within curly braces. The variable name in JavaScript could be any valid identifier or string and value could be constant or JavaScript expression. Since JavaScript is dynamically typed, variables are declared using var keyword(see above example). You could also declare variables without the var keyword but that would mean that scope of the variable is global.

Note: This post will not look into scoping and namespace but it’s really important to have that knowledge especially when building frameworks.

Here is another example from Prototype.js

var Prototype = {
Version: '1.6.0', //Constant value
Browser: {
IE: !!(window.attachEvent && !window.opera), //javascript expression
Opera: !!window.opera,
WebKit: navigator.userAgent.indexOf('AppleWebKit/') > -1,
Gecko: navigator.userAgent.indexOf('Gecko') > -1 && navigator.userAgent.indexOf('KHTML') == -1,
MobileSafari: !!navigator.userAgent.match(/Apple.*Mobile.*Safari/)
},
...
...
};


Here in this example we are creating Prototype object using object literal syntax and "Version" is one of the property of the Prototype object with value of "1.6.0".

Objects are created when JavaScript interpreter evaluates the object literals. Once the object is created you can access its properties using the "." operator.
For Example

myJSONObject.ircEvent (produces: "PRIVMSG")
Prototype.IE (produces: true or false based on browser)

We should note that object properties doesn't have to be boolean or string, it could be another object as well. In the case of Prototype, "Browser" is a property and it is also an object with properties like IE, Opera, Webkit etc.

In the re-introduction section we discussed about the nature of the prototype based languages and how it encourages programmers to evolve their existing objects. In JavaScript we can create properties when we need them by simply assigning value to it.

Example from scriptaculous/unittest.js

var Test = {} //creating a empty object
Test.Unit = {}; //creating a new Unit property to Test object
// security exception workaround
Test.Unit.inspect = Object.inspect; //adding a new property to Unit object

In the above example we are creating an empty Test object without any property but moment we do Test.Unit = {}, it is creating a new property called Unit inside the Test object. We could also delete prop-erties from object on the fly using "delete" operator.

Earlier we looked how objects are created and how to access object properties using "." operator but we haven't formally defined JavaScript object yet.

In JavaScript, object is an unordered collection of properties. What it means is, we can loop through the object properties and access them as associative array using [] operator. This was very confusing to me before I understood how JavaScript objects are defined.

Example from scriptaculous/effect.js

//inspect property inside some object
inspect: function() {
var data = $H(); //some function, yes $ is allowed
//looping through all the properties defined inside this object
for(property in this)
if (!Object.isFunction(this[property])) //here object property is accessed as an array.
data.set(property, this[property]);
return ...;
}

Using "[]" operator has one advantage over the "." operator because the name of the property is expressed as string and could be manipulated in runtime.

someObject["someProperty" + "_en"]

Before we leave associative arrays, I like to mention that JavaScript also has a concept of array and it is defined as an ordered collection of values. Since JavaScript is dynamically typed we could create array with many different types, here is an example

var someArray = [ "this", "is", [1], way] //way is some variable already defined

JavaScript also defines additional methods like reverse, sort, slice, push, pop and others (complete list of methods and properties) for ar-ray. If we take a look at the list of methods it could be surprising to see methods like push and pop as if we are using stack instead of ar-ray. JavaScript arrays could be used as stack by pushing an element and popping an element.

Example from prototype.js

…
keys: function(object) {
var keys = [];
for (var property in object)
keys.push(property);
return keys;
},
values: function(object) {
var values = [];
for (var property in object)
values.push(object[property]);
return values;
}
…

In this code snippet we are pushing property names to keys array and pushing property values to values array.

Its quite similar to Ruby where array could also be used a stack but unlike Java where there is separate class for Array and Stack. It al-most feels like dynamic languages have their own design principles separate from other languages. Advocates of Humane interfaces would argue that there could be multiple usage of an array, as an array or as a stack and hence we see methods like push and pop.

Learning is your bottleneck

"Our civilization runs on software." It is therefore a tremendous privilege as well as a deep responsibility to be a software developer
Bjarne Stroustrup

I just started reading about “Agile Adoption Patterns” book and I must say I am already impressed. It starts with “Learning is the bottleneck” chapter that example how that is main problem with any project and how agile helps mitigate that risk.

Why learning is important? Because we cannot know everything about a project even you are doing software development for 20 years and unknowns are risk in projects. No two projects are same. Even if you end up using same technology set, your teammates are different or maybe it is a new organization. So we have to learn things as we go (no assumptions doesn’t help) by iterations.

Why it’s a bottleneck? Software project failures have a lot in common with airplane crashes. Just as pilots never intend to crash, software developers don't aim to fail. It is amazing to look at history of software failures, here are some reasons why software project fails

• Unrealistic or unarticulated project goals
• Inaccurate estimates of needed resources
• Badly defined system requirements
• Poor reporting of the project's status
• Unmanaged risks
• Poor communication among customers, developers, and users
• Use of immature technology
• Inability to handle the project's complexity
• Sloppy development practices
• Poor project management
• Stakeholder politics
• Commercial pressures

And it is quite interesting how all these reasons are somehow tied to not having a negative feedback cycle so that teams can make smaller mistakes and learn from them.
Only if we could have opportunity to rectify our oversights sooner!

Yes we can, we need a software development process that allows to us learn from our mistakes. The iterative nature of an agile process helps us to make small mistakes and fail fast helps us to make necessary corrections. And if we don’t pay enough attention to them we might end up as another failed project in the list. But wait is it only process that could save us from the misery? No it is always “people over process”. If we are not paying attention to our mistakes no process can help. We need people who are adaptive and ready to learn, “we need Cathedral builders not stonecutters”

Code Coverage is a negative metrics tool

This is something we forgot on our way with coverage tools over the years and I think, needs to be restated. Its really not important how much code is covered by your unit tests(assuming they are good) but rather which parts of code is not been covered by your tests.

Literate Programming to DSL

Literate programming is phrase coined by Donald Knuth to describe a certain approach of developing computer programs from the perspective of a report or prose. The basic idea is to create self-documenting code. The programmer should write code as if he or she is explaining to another programmer how the code will be executed by the machine, like a literature. Knuth implemented literate programming through the WEB system of structured documentation. It uses Tex for documentation and C for executable code.

Sounds like a awesome idea but I wonder why it didn't take world by storm like DSL. According to the interview recently published, Knuth thinks only small percentage of programmers are good at writing, hence never been adopted that well. What do you guys think? Now it keeps me wondering how good are we going to be with DSL, doesn't it requires same skill. It is not easy to come up with good succinct, well readable DSL. Do we need new skills to be good at it?

One beauty of DSL is, its domain specific and sometimes its really small and succinct, so no need to have extra documentation. Sometimes its important to have documentation if you allow your business users to play with your DSL but having them integrated part of the code could make it bit clumsy and verbose. And like any other type documentation it still have the same inherited problem of keeping up with code changes (maybe having them integrated might help). Now with languages like Ruby and Groovy its possible to write human readable executable code without any additional documentation.


Update: Nice blog entry from PragDave. Addresses few core issues associated with DSLs
http://pragdave.blogs.pragprog.com/pragdave/2008/03/the-language-in.html

Big enterpise deal going bad

http://www.itworld.com/App/670/waste-management-sues-sap-080327/

autotest for java

I just hacked out a java version of autotest plugin for eclipse (of course inspired by the ZenTest).

No more running JUnit tests manually, this plugin will detect code changes and run JUnit tests of that Java project. You can download and install this plugin from http://code.google.com/p/autotest4j/

Current version of plugin only runs with eclipse 3.3 and I need to do some clean up and fixes before I can checkin my source code. Watch out for some changes I am planning for next version.

Enjoy...

rss to json

Here is a nice yahoo pipe if you want to process rss feed from you web pages
rss to json

Scala: Surprise Part oNe

Ruby is a language of least surprises but I am not sure whether I can say that for Scala yet. I have already encountered some of them, here is one example

You can define method without '()'

def height: Int = 0 //This is a valid method

def width(): Int = 0 //This is valid too

Now you can invoke 'width' method with and without parenthesis but you cannot do that for height method.

SURPRISED


Update: The Scala convention is, use '()' for parameter less method if it has side effects otherwise don't