Ezra's Research

Here's a stimulating article: Why Most Published Research Findings Are False by John P. A. Ioannidis (PLoS Medicine, 2005). It focuses on research that aims to find statistical relationships in data, and asserts that most such relationships claimed in the literature are in fact false. Distilling the discussion, I find these compelling reasons why it would be so:

• standard tests of "statistical significance" are taken as proof of a proposition,
• there is bias in experimental design & interpretation,
• researchers and journals prefer positive results,
• experiments tend not to be independently reproduced.

This last point is particularly damning—few things are more essential to the scientific method than reproducible experiments, yet the article blithely says (and I readily believe) that most biomedical studies are not reproduced. In fact, the competitive publication cycle works against this: merely to confirm an existing result is not very publishable; to contradict an existing result may be publishable, but this means, as Ioannidis notes, that there can be an alternation of positive, then negative, then positive, then negative results on a particular question, as each team becomes interested in upsetting the last published result. Far from amassing independent evidence on a question, this is just another selection bias that works against the scientific process.

Interestingly, the article is wholly unscientific. Without stating its assumptions, it works these assumptions through to conclusions. Along the way, it presents a bunch of formulas, which add the gloss of analysis to what is essentially a work of persuasive writing—but I don't buy the formulas, which include unobservable (and perhaps ill-defined) quantities such as "the ratio of true relationships to no-relationship pairs in a field" and "the false-negative error rate." (How amusing would it be if this article were a false research finding?) But methodology aside, I do believe it: that many, it not most, published research findings are false.

I'd be interested to see someone look at the issue in other kinds of fields—fields that aren't quantitative, for example. In the field of programming languages, people make a lot of claims that are justified by proofs. How often are these proofs actually correct, I wonder? And how often are the claims correct? Moreover, much PL research is not even "claim-based," as such. Many papers simply present a feature or technique and tout its virtues—and don't make falsifiable claims, at all. And often, this is the most valuable research: someone tried something and described the experience. We can learn from others' experience, even without proven claims.

How do we assess the value of a research field, such as that of programming languages? How do we know when we're doing a good job?

Posted by Ezra elias kilty Cooper at 1:11 PM | Permalink | Comments (3) | TrackBacks (0)

Posted by Ezra elias kilty Cooper at 11:31 AM | Permalink | Comments (2) | TrackBacks (0)

• A declarative expression language notation is not specified in (X)HTML. As a result developers have to code dynamic aspects of the user interface by means of server side scripts or Javascript DOM [DOM] manipulation.
• (X)HTML classic forms [Forms] are severely limited in terms of typing information. For example, It cannot be specified that an input element accepts a date, a number or a patterned text. Furthermore, due to limitations in the submission formats, applications have to convert everytime, typically at server side, strings to the correct type expected by the application logic.
• Lack of data binding mechanisms. Data binding is a technique that allows to associate data between the model and user interface components in a declarative way. Developers typically write server side scripts or Javascript AJAX-based code to populate user interface components with data. As a result, in (X)HTML-based user interfaces there is no clear separation between the model, the view and the controller.

Posted by Ezra elias kilty Cooper at 3:09 PM | Permalink | Comments (2) | TrackBacks (0)

DrScheme is a very good app for writing Scheme code. It's much better than any other that I've used, or am aware of. But I do have some complaints; I'm lodging them here for later reference.

1. When I get an error that my parens don't match, the blinky-paren-matching feature is turned off! (The error is highlighted in pink, and paren-blinking is turned off in the pink region.)

2. In the REPL, there's no keystroke for "execute this": instead I have to go to the end of the expression, then hit Return. (Something like Shift-Return would do the trick.)

3. In the REPL, there's no easy way to retrieve the last command for editing. (In command shells, this is usually just Up-arrow. Is there any reason not to use that here. Surely I'll never need to use the keyboard to select random interpreter output, if that's what it's being saved for. I'd be quite happy, though with Ctrl-Up-arrow.)

4. Cmd-Left-arrow (on the Mac) takes me to a point before the interpreter prompt. It's inconceivable I could ever want to go there. Make it take me to the beginning of my expression, just after the prompt.

5. The Modifier-Arrow combinations don't work at all in the Help window's input field, and neither does double-clicking to select by words.

6. The dock/undock feature for "palette" windows is not very compelling, but it does take up space and confuse matters. On the Mac in particular, the Find/Replace palette comes up as a "sheet," extending from the top of the window, which is entirely the wrong mode, for at least two reasons. First, because it's modal: it doesn't let me interact with the underlying window. Second, because it sometimes obscures the very text I'm looking for. Pretty much every text editor on the Mac does Find/Replace with a floating window—why not do it that way? Perhaps the platform-independence of DrScheme makes this hard to implement. In that case, it might be okay to use an ordinary (non-floating) window: one that can be hidden by other windows. In that case, there is a question of which definitions window will be searched. Possible right answers are: (1) the one that was frontmost when I opened this Find/Replace window, or (2) whichever one is frontmost when I hit the "Find" button.

7. The behavior of Ctrl-arrows and Alt-arrows are reversed, IMHO. On the Mac, Option (Alt) always means "move by words" and Ctrl usually means something else, appropriate to the context. DrScheme's Ctrl-arrows move by word, while Option-arrows move by expression; this should be the other way around to keep with the Mac conventions. I wonder if other Mac users agree?

Posted by Ezra elias kilty Cooper at 6:37 PM | Permalink | Comments (2) | TrackBacks (0)

Gripes with OCaml:

• Types that are declared in an interface file also need to be redeclared in the implementation file.
• let is stranger word than fun (as in SML) for declaring functions
• OCamlMakefile doesn't detect dependencies if the interface file is not mentioned in SOURCES; "X and Y have inconsistent assumptions"
• No built-in "compose" combinator, can't use o (as in SML) or . (as in Haskell)
• No before form as in SML (very useful when debugging, checking return values)
• a record type cannot be given directly within a variant constructor declaration (e.g. Foo of {field : string}).
• Syntax errors are often unhelpful (Syntax error)
• Never helpfully points out that you might've partially applied a function when you meant to fully apply it (Haskell does help in this way!).
• Interactive shell is very difficult to use:
  • It is picky about terminating each expression with exactly two semicolons.
  • There are lots of special commands for loading modules (open FooModule is never enough) and they have different conventions: for example, some begin with # and require an exact filename. I never know what file needs to be #loaded.
• The build system detects when a module's interface has changed, but forces programmer to do something about it.
• The built-in regexp interface is very clunky:
  • Str.quote needlessly returns a string instead of a regexp, requiring me to then call Str.regexp whenever I use it.
  • Str.string_match needlessly requires an integer argument designating the index at which I'd like to start matching. This should, of course, be an optional argument or a distinct form of the function (Perl seems to have gotten along okaytaken over the world without a silly index argument on regex matching).
• The Map module for finite maps provides a map function for transforming the domain of the map, but doesn't provide a way to transform the keys.

Posted by Ezra elias kilty Cooper at 7:39 PM | Permalink | Comments (2) | TrackBacks (0)