and a little interface for it. this is trying to spell the words using phonetic information (using a sequence-to-sequence neural network), the temperature parameter basically controls how the probabilities are distributed (at low temperatures, only the most likely characters are generated according to the information in the model; at higher temperatures, any character might be generated)

I need to stop playing with this, I have other stuff to do geez

still at work on this english nonsense word vae. here are some nonsense words sampled from the latent space of the latest trained model...

twidle
tuppilled
entedrul
tremdpobe
chominsbow
gripkan
dirquineus
dudenowed
rostore
kigan
nedermotta
sastors
lielandi
zessermas
ricknest
chated

these are generated by feeding the decoder with normally-distributed random numbers. pretty happy with how they all seem like jabberwockian-yet-plausible english words

by contrast, results of feeding normally-distributed random numbers into the decoder on the RNN without the VAE:

flfingeng
aughums
alohondism
h's
h's
autabovag
akeleghear
h's
alliltalles
barngnong
h's
mook
shewstlatscreth
huthure
chelthart
h's

not as good! which is encouraging, since it shows that the VAE model does actually have a "smoother" space than the non-VAE model.

(I have to admit that when I started this project I was like, "why do you even need a variational autoencoder, if just plugging random vectors into the decoder was good enough for jesus it's good enough for me," but there really is something magical and satisfying about being able to get more-or-less plausible generated results for basically any randomly sampled point in the distribution)

progress: at 50 epochs, even w/KL annealing, 32dims is not enough for the VAE latent vector to represent much of anything. leads to reconstructions that are probably just the orthography model doing its best with next-to-noise, but sometimes amusing, e.g.

cart → puach
liotta → pinterajan
intellectually → aching
capella → pellaka
photometer → augh
sympathizer → disteghway
butrick → jorserich
botha's → szine
clayman → tsantiersche
sparkles → trenlew
calamity → muliss
thermoplastic → tphare

(posted this mainly because "butrick → jorserich" seems like something mastodon people would like, e.g. "my name is Butrick Jorserich, follow me at jeans.butrick.horse/@rich")

apparently the trick to training a VAE w/annealing is to *never* let the KL loss go below the reconstruction loss. otherwise you get beautifully distributed, wonderfully plausible reconstructions that have almost nothing to do with your training data, i.e., "allison" becomes

uszecuin
auruse
lin-timer
ellefleigh
carmist
achubar
alsaa
houghtrodhan
ascear
edding
earpugh
ihtioz

exploring the latent phonetic nonsense space around "typewriter"—using the best model I've managed to train yet (100 epochs on HPC, managed to keep the reconstruction loss fairly low while also getting some semblance of a low KL loss)

using the phonetic VAE to interpolate between US state names in a grid

decoding the same underlying vectors from the VAE using the french spelling model, for some reason, sure, whatever

visualizing the vector in the latent phonetic space while interpolating between "abacus" and "mastodon." (this is after inferring the latent vectors via orthography->phoneme features->VAE). I just arbitrarily reshaped the vectors from (1, 1, 128) to (8, 16), so the 2d patterns are arbitrary. still interesting to see what it's actually learning!

exploring the phonetic space by just setting large chunks of the dimensions to arbitrary values

generating random magic words by adding progressively more noise to "abracadabra"

abrakadabra
abrakadaba
abracadabra
abracadabara
abrocabreda
abhricada
aereakademada
azhikaradha
ebrakadabad
abrrimaluding
avvirivasava
avrvakdavulusus
zzbhimizzzzzhizzzhr
vbp-bupparkabpappap
ablamabalabtab
adtddtakaka'sk
abbhxababblix
nayaaaaaaayaadaadaa

@aparrish terich sounds like a nice place to live

@aparrish some of these rows and columns make good nonsense poetry

@aparrish brb, setting a fantasy novel in Oache-Gnang

@aparrish Have you tried doing t-sne or some other kind of clustering on the dimensions?

@mewo2 i haven't but i probably should! i've just been going by my gut and the kl loss of the vae model to tell me that it has a normal(ish) distribution but it would be nice to have some visual evidence

@aparrish vbp-bupparkabpappap

so THAT'S what my neighbour's yappy dog is doing

@aparrish

> generating random magic words by adding progressively more noise to "abracadabra"

You didn't *quite* end up with "Avada Kedavra", but pretty close!

@aparrish

until, finally, somehow, mysteriously

it turned out

that Fred Rogers was right all along:

p'lease

@aparrish

alternate reply:

bompbadabompadangadingdongadongadingdangBLUEMOOOOOO

@aparrish
Stop taking the above if you experience any adverse reactions...

@aparrish

nayaaaaaaayaadaadaa
ablamabalabtab
ebrakadabad

it's the circle of life
and it moves us all

@aparrish Heh apparently we're in Unsong. Trying to bruteforce generate words for Kaballah? :P

@aparrish the third one is just "abracadabra" though! Did the noise randomly all turn out close to zero on that one?

@aparrish
vbp-bupparkabpappap
she's my baby
vbp-bupparkabpappap
i don't mean maybe

@aparrish Are you looking for reasons to have States named after you? Parishonde. Parrison.

@aparrish I've always wanted to visit Pariss, and now I know where it is.

@aparrish 1) i love this 2) do you have any idea why theres exactly one fake state name name with a hyphen in that chart? It seems like if it was an option, there should be more, but

@Satsuma hyphens are in the training data vocabulary but don't happen very frequently. I think it was just happenstance in this case that the spelling model picked that character in that position

@aparrish Is california missing its second 'i' due to something about the spelling model?

@JoshGrams the VAE doesn't perfectly encode phonetics, and the spelling model is doing its best guess with the results—plus the spelling model has a bit of randomness built into the inference process. so things like that are kind of an expected part of the process in this case!

@aparrish Ah, got it. I saw some other -nia words so I thought it might be *something* like that. Thanks!

Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!