Mainly Tech projects on Python and Electronic Design Automation.

Monday, February 04, 2013

No Attempt At Rigorous Proof

I had flu and so read/skimmed a lot more than usual this week and came across two conference videos that basically said that there is very little engineering and science in the books, papers, and conference proceedings of the software industry out there.
Many famous and/or well-read authors show little or no scientific or engineering rigour in the formulation of their conclusions. They are basically saying "Do this 'cos I said you would get better results".

So many blog posts are exactly the same. After reading Be neat, and the rest will follow I was moved to make just that point in the comments. It's not that I don't agree or disagree with what they are saying, it's just that I came to realize what was missing from the blog post.

Now me writing long comments of this kind of thing to blogs:


Hmm. Where's your evidence and scientific method? What you say may well be true but you have no science to back it up. You might as well be harping on about a flat earth!
  • Do you have an impartial and reproducible tool to determine neatness for example?
  • Do others perception of neatness coincide with yours?
  • Any emperical statistics to back up your statement?
  • Where is the graph of neatness vs "cost" to back up what you are saying?
You might be right but you have not shown scientifically that you are right. You could be selling snake oil!

... Is not snappy and is not likely to grow so What I think is needed is an anagram.

Naarp!

If you think that a post is all about trying to sell you the best way to do something when programming, but the post makes no attempt to show you reproducible evidence supporting their claim then just slap a

    NAARP!

In the comments, where NAARP stands for:

No
Attempt
At
Rigorous
Proof


If spoken, Naarp is pronounced as if spoken by a village idiot/country bumpkin. I was thinking especially of the large guy in Hot fuzz who says Yaarp for yes, and of Simon Pegg thinking up Naarp when impersonating him to mean no.

Hopefully it can show that people can recognise the lack of scientific method in the industry and maybe nudge authors to be aware that their audience is now aware of this too.

- Peace.

5 comments:

  1. NAARP is a criticism applicable to mathematics. It does not apply to the scientific method.

    No scientific theory can be rigorously proven. The scientific method requires only that theories are consistent with available experimental/observational data. That is, it is not possible for scientific law to be deduced from scientific data. For example, Newton's Laws of Motion are only applicable for masses moving much less than the speed of light. Only at low speeds (compared to light) are these Laws consistent with observation.

    It is not rhetoric that is missing from most software engineering literature -- it's data. There is an old aeronautical engineering saying: "Theory without data is just opinion."

    ReplyDelete
  2. A. Like gmcrews mentions, NAARP is inappropriate.

    B. Google up some empirical software engineering and mining software repositories. Such as http://www.americanscientist.org/issues/pub/empirical-software-engineering

    ReplyDelete
  3. @Paddy3118: you are right that all of this is relative to my subjective experience. I have never claimed that any of this is backed up by scientifically valid research and data. It is highly unlikely that I will ever bother performing actual experiments to back up these observations and conclusions. I'll leave that to people in academia or people who want to sell books.

    It is entirely possible that my observations are wrong. My data set might be skewed in some way or there may be some observation bias. I consider the above to be a rather conservative conclusion based on personal experiences from 25 years of software development.

    As for objective measures of neatness: I don't think they exist. I'm not even sure they are needed to support my observations because what if my neatness measure is correlated with the positive effects and yours are not?

    And as for a neatness/cost graph....I don't expect it to be that simple but I find it intriguing that you do.

    (And finally, as for snake oil: take a look at the history of what has been written about, for instance, pair programming and how "scientific" the software engineering community was in dealing with it. Then read Feynman's commencement address "Cargo Cult Science" and ponder whether you would be able to tell science from pseudoscience)

    ReplyDelete
  4. Good points, but if we waited for rigorous scientific proof before trying anything new ourselves, we wouldn't be very active. When someone suggests a practice that "intuitively" appeals to me, I'll give it a try. If it works for me, then I'll add it to my toolbox. Otherwise I'll chuck it and move on.

    ReplyDelete

Followers

Subscribe Now: google

Add to Google Reader or Homepage

Go deh too!

whos.amung.us