and a little interface for it. this is trying to spell the words using phonetic information (using a sequence-to-sequence neural network), the temperature parameter basically controls how the probabilities are distributed (at low temperatures, only the most likely characters are generated according to the information in the model; at higher temperatures, any character might be generated)
I need to stop playing with this, I have other stuff to do geez
still at work on this english nonsense word vae. here are some nonsense words sampled from the latent space of the latest trained model...
these are generated by feeding the decoder with normally-distributed random numbers. pretty happy with how they all seem like jabberwockian-yet-plausible english words
by contrast, results of feeding normally-distributed random numbers into the decoder on the RNN without the VAE:
not as good! which is encouraging, since it shows that the VAE model does actually have a "smoother" space than the non-VAE model.
(I have to admit that when I started this project I was like, "why do you even need a variational autoencoder, if just plugging random vectors into the decoder was good enough for jesus it's good enough for me," but there really is something magical and satisfying about being able to get more-or-less plausible generated results for basically any randomly sampled point in the distribution)
(posted this mainly because "butrick → jorserich" seems like something mastodon people would like, e.g. "my name is Butrick Jorserich, follow me at jeans.butrick.horse/@rich")
in which I accidentally leave off the "end" token when predicting spelling from sound, and it just keeps on spelling until it's ready to stop
remarkable → remarymarkamal
wysiwig → irzerwizkian
bemuse → bismebishews
unenforceable → unofironsfinars
shutters → shurtsithaters
capstick → capstickapsitk
vittoria → viltovitria
beilenson → billabinsancin
peers → pieespianes
paste → past-pasest
excitable → exexaitabile
phibro → fib-to-birbo
croney → crainkrine-y
tangle → tangitangle
"how doth the little crocodile improve his shining tail and pour the waters of the nile on every golden scale" → neural network spelling by sound but with probabilities of [aeiou] zeroed out → "hv d'th thy lyttl crch-dykly mpr h's shynnyng thyl hnyd ph thy whytrs f thy nyl hwhn avry ghqlynd schqly"
apparently the trick to training a VAE w/annealing is to *never* let the KL loss go below the reconstruction loss. otherwise you get beautifully distributed, wonderfully plausible reconstructions that have almost nothing to do with your training data, i.e., "allison" becomes
going back to the regular seq2seq networks, I'm trying to do some quantitative evaluation. the phoneme features to orthography model gets... ~60% of words wrong, and ~12% of letters wrong (working on samples a few thousand words from cmudict), but its guesses seem... reasonable? not sure how to talk about this
this is showing the original words on the left, and the "sounded out" words from the model on the right. the model I trained going the other way (orthography to phoneme features) performs pretty close to published baselines for similar grapheme-to-phoneme models (phoneme error rate = ~7%, word error rate = ~30%), but there are comparatively fewer papers about phoneme-to-grapheme models, so I'm not sure if this is "good" or not
it might just be the case that sounding out a word based on its spelling relies only a little bit on context, but spelling a word from how it sounds relies on context a lot. ("context" in this case meaning anything other than, like, a character-based language model of individual words)
my instinct is that if you trained on the phonemes AND a distributional word vector, you'd get pretty good accuracy on this task! but someone's probably already done that and/or that's a project for another day
was curious to see if my model could produce reasonable portmanteaux. steps: translate spelling to sound, predict the sound-spelling model's hidden state for 2 words, average those states & decode
breaker + nylon → breichor
underwear + futility → unterilie
intolerance + homer → honepheren
Cabot + coyote → caibott
bonus + boasting → boensing
by-election + basin → baisention
demonstration + scissors → cesserser
volcano + baron → balano
tiger + panther → paighter
sharpness + hardship → sharpship
(mathematically this basically just works out to averaging the phonetic features of both words at each timestep and telling the sound-spelling model to do its best. which is of course not how portmanteau actually work—usually we try to find a useful point of similarity in two words, then cut from the first word to the second at that point. but I like to imagine that you could make portmanteau by just smooshing two words together like two skittles)
visualizing the vector in the latent phonetic space while interpolating between "abacus" and "mastodon." (this is after inferring the latent vectors via orthography->phoneme features->VAE). I just arbitrarily reshaped the vectors from (1, 1, 128) to (8, 16), so the 2d patterns are arbitrary. still interesting to see what it's actually learning!
generating random magic words by adding progressively more noise to "abracadabra"
@aparrish Sounds like a distant branch of the Joestar family tree from JoJo's Bizarre Adventure
@aparrish As the Most Interesting Man in the World, I don't always wear jorts. But when I do, I wear Butrick Jorserich.
@aparrish evollove it!
Tangitangle was always my favourite pokemon
@aparrish reverb (and renoun, etc)
Ooh now I want a bot that runs posts through this
@aparrish this reads like how a super-low bitrate mp3 sounds
@flussence 100% exactly what I'm going for 😁
@aparrish I'd buy that as a lost snippet from Beowulf on first examination
@aparrish it snuck a single ‘a’ in there! Th mshjn rhblyhn bhgnz
@aparrish A lyttl Chaucerian English?
@aparrish what’s my name what’s my name
@courtney according to this, it's Che-cotort-an-ratily Statstaintangent
@aparrish thank you this is perfect
@aparrish weeeeelllll if somebody ain't FANCY
@aparrish oh, that's how I was already pronouncing it
@aparrish mama say mama sa mammasavinstawn
@aparrish Prof. Dodgson would have laughed at that. Try the program on Jabberwocky.
"I know how to spell banana I just don't know when to stop"
what hath the halting problem wrought?
@aparrish that jargon is Star-Trek-level!
I think you should adopt "houghtrodhan" as your secret identity when in Britain.
@aparrish after composing a love poem on your Typeriker
@aparrish These words are all perfect. And now let me get back to my micepotor to compose some letters.
@aparrish btw please just apply these transforms to an entire book for a half hour nanogenmo this year
@aparrish those are amazing. tbh let's just normalize english spelling by making sure a small neural net can map between spelling and phonemes. that's probably a more justifiable plan than any of the earlier attempts?
@aparrish It feels like these are potential spellings of English words, but not *common* English words? I wonder if weighting the training words by frequency would push it more towards "standard" spellings?
@mewo2 I have been meaning to try that. but some of these already feel like more "standard" spellings, or are just attested alternate spellings (morice -> morris, whitacre -> whitaker), so I'm not sure if it would help
@aparrish They feel very right to me? Like, when I say the word on the left (those times that I recognize the word on the left), I can tell very easily why it might read that pronunciation the way it spells it.
I know far too little linguistics to put that into words, but yeah.
@aparrish terich sounds like a nice place to live
@aparrish some of these rows and columns make good nonsense poetry
@kragen that's the idea!
I would not mind visiting califania
@aparrish brb, setting a fantasy novel in Oache-Gnang
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!