What happens if you do a survey on Mechanical Turk asking programmers what semantics they expect of their languages? has the answer! Direct link to PDF:


Their overall conclusion: users were not consistent with each other, nor with themselves, and sometimes favoured semantics that are widely avoided in modern language design, such as dynamic scoping.

For example, whilst 39% expected that trying to access a parameter outside of a function would always error, 17% thought that the first code snippet would run without erroring!

The questions about truthiness were even more fun. People expected 0 to be falsy nearly as much as they expected true to be truthy. They were split on "0".

A remarkable 7% thought a literal true was falsy!

The results for numbers seemed much more defensible to me. People either expected exact fractions or integer arithmetic (both nicer to work with than floating point). It was hard to separate the printed representation from the internal representation in the question.

Results for scope was much more messy.

28% presumed dynamic scope (probably assuming globals or JS-style lifting of undeclared vars), 16% presumed static scope, but 19% presumed a mix!

I think the syntax influenced the question: it'd be interesting to see let blocks measured.

One result I found intriguing: 30% expected aliasing (i.e. call-by-reference) and another 30% did not (i.e. call-by-value) for records! I would have expected call-by-value to be lower (I've been caught out by this in codebases).

The authors are candid about the challenges of their approach: Mechnical Turk can be noisy, syntax similarity to concrete languages can influence responses, and demographics of respondents could make a big difference.

The results were certainly different to my expectations.

Overall, it's an interesting paper that demonstrates that you cannot entirely crowdsource language design. I didn't cover everything, so I'd recommend reading it for yourself (especially if you're interested in PL design)!

this part really makes me want to take programs made by people in intro to programming courses and make compilers / interpreters that make them valid. wonder what that'd look like, might be very cool

@wilfredh This would explain widely avoided Lisp implementations like Newlisp and Bee Lisp...

Sign in to participate in the conversation

Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. This server is run by the main developers of the Mastodon project. Everyone is welcome as long as you follow our code of conduct!