and a little interface for it. this is trying to spell the words using phonetic information (using a sequence-to-sequence neural network), the temperature parameter basically controls how the probabilities are distributed (at low temperatures, only the most likely characters are generated according to the information in the model; at higher temperatures, any character might be generated)

I need to stop playing with this, I have other stuff to do geez

still at work on this english nonsense word vae. here are some nonsense words sampled from the latent space of the latest trained model...

twidle
tuppilled
entedrul
tremdpobe
chominsbow
gripkan
dirquineus
dudenowed
rostore
kigan
nedermotta
sastors
lielandi
zessermas
ricknest
chated

these are generated by feeding the decoder with normally-distributed random numbers. pretty happy with how they all seem like jabberwockian-yet-plausible english words

by contrast, results of feeding normally-distributed random numbers into the decoder on the RNN without the VAE:

flfingeng
aughums
alohondism
h's
h's
autabovag
akeleghear
h's
alliltalles
barngnong
h's
mook
shewstlatscreth
huthure
chelthart
h's

not as good! which is encouraging, since it shows that the VAE model does actually have a "smoother" space than the non-VAE model.

(I have to admit that when I started this project I was like, "why do you even need a variational autoencoder, if just plugging random vectors into the decoder was good enough for jesus it's good enough for me," but there really is something magical and satisfying about being able to get more-or-less plausible generated results for basically any randomly sampled point in the distribution)

progress: at 50 epochs, even w/KL annealing, 32dims is not enough for the VAE latent vector to represent much of anything. leads to reconstructions that are probably just the orthography model doing its best with next-to-noise, but sometimes amusing, e.g.

cart → puach
liotta → pinterajan
intellectually → aching
capella → pellaka
photometer → augh
sympathizer → disteghway
butrick → jorserich
botha's → szine
clayman → tsantiersche
sparkles → trenlew
calamity → muliss
thermoplastic → tphare

Follow

(posted this mainly because "butrick → jorserich" seems like something mastodon people would like, e.g. "my name is Butrick Jorserich, follow me at jeans.butrick.horse/@rich")

in which I accidentally leave off the "end" token when predicting spelling from sound, and it just keeps on spelling until it's ready to stop

remarkable → remarymarkamal
wysiwig → irzerwizkian
bemuse → bismebishews
unenforceable → unofironsfinars
shutters → shurtsithaters
capstick → capstickapsitk
vittoria → viltovitria
beilenson → billabinsancin
peers → pieespianes
paste → past-pasest
excitable → exexaitabile
phibro → fib-to-birbo
croney → crainkrine-y
tangle → tangitangle

"how doth the little crocodile improve his shining tail and pour the waters of the nile on every golden scale" → neural network spelling by sound but with probabilities of [aeiou] zeroed out → "hv d'th thy lyttl crch-dykly mpr h's shynnyng thyl hnyd ph thy whytrs f thy nyl hwhn avry ghqlynd schqly"

"How doth the little srurbumbered improve his shining pearple and pour the borbirpers of the mrilmer on every golden sprarple"

(adding bilabial and rhotacization features to sounds of *just the nouns* when decoding from phonetics to spelling)

inferring spelling from phonetic feature sequences zoomed along the timeseries axis. (basically, smooshing and stretching the sound of the word and getting the neural network to try to spell out the sound)

in case you're wondering, if you scale the sound of "mastodon" by 4x, it spells "mammasavinstawn"

my name at 4x: Alarlaslilliance Pempereterriashi

the problem with this gug-aradamptling project is that I can't stop playing around with it long enough to write about it

apparently the trick to training a VAE w/annealing is to *never* let the KL loss go below the reconstruction loss. otherwise you get beautifully distributed, wonderfully plausible reconstructions that have almost nothing to do with your training data, i.e., "allison" becomes

uszecuin
auruse
lin-timer
ellefleigh
carmist
achubar
alsaa
houghtrodhan
ascear
edding
earpugh
ihtioz

@aparrish Sounds like a distant branch of the Joestar family tree from JoJo's Bizarre Adventure

@aparrish As the Most Interesting Man in the World, I don't always wear jorts. But when I do, I wear Butrick Jorserich.

@aparrish
Ooh now I want a bot that runs posts through this

@aparrish it snuck a single ‘a’ in there! Th mshjn rhblyhn bhgnz

@courtney according to this, it's Che-cotort-an-ratily Statstaintangent

@aparrish oh, that's how I was already pronouncing it

@aparrish Prof. Dodgson would have laughed at that. Try the program on Jabberwocky.

@aparrish
"I know how to spell banana I just don't know when to stop"

@aparrish
I think you should adopt "houghtrodhan" as your secret identity when in Britain.

Sign in to participate in the conversation
Mastodon

Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. This server is run by the main developers of the Mastodon project. Everyone is welcome as long as you follow our code of conduct!