and a little interface for it. this is trying to spell the words using phonetic information (using a sequence-to-sequence neural network), the temperature parameter basically controls how the probabilities are distributed (at low temperatures, only the most likely characters are generated according to the information in the model; at higher temperatures, any character might be generated)
I need to stop playing with this, I have other stuff to do geez
still at work on this english nonsense word vae. here are some nonsense words sampled from the latent space of the latest trained model...
these are generated by feeding the decoder with normally-distributed random numbers. pretty happy with how they all seem like jabberwockian-yet-plausible english words
by contrast, results of feeding normally-distributed random numbers into the decoder on the RNN without the VAE:
not as good! which is encouraging, since it shows that the VAE model does actually have a "smoother" space than the non-VAE model.
(I have to admit that when I started this project I was like, "why do you even need a variational autoencoder, if just plugging random vectors into the decoder was good enough for jesus it's good enough for me," but there really is something magical and satisfying about being able to get more-or-less plausible generated results for basically any randomly sampled point in the distribution)
progress: at 50 epochs, even w/KL annealing, 32dims is not enough for the VAE latent vector to represent much of anything. leads to reconstructions that are probably just the orthography model doing its best with next-to-noise, but sometimes amusing, e.g.
cart → puach
liotta → pinterajan
intellectually → aching
capella → pellaka
photometer → augh
sympathizer → disteghway
butrick → jorserich
botha's → szine
clayman → tsantiersche
sparkles → trenlew
calamity → muliss
thermoplastic → tphare
apparently the trick to training a VAE w/annealing is to *never* let the KL loss go below the reconstruction loss. otherwise you get beautifully distributed, wonderfully plausible reconstructions that have almost nothing to do with your training data, i.e., "allison" becomes
visualizing the vector in the latent phonetic space while interpolating between "abacus" and "mastodon." (this is after inferring the latent vectors via orthography->phoneme features->VAE). I just arbitrarily reshaped the vectors from (1, 1, 128) to (8, 16), so the 2d patterns are arbitrary. still interesting to see what it's actually learning!
generating random magic words by adding progressively more noise to "abracadabra"
@aparrish The last is what conjures the nayan cat.
@aparrish aardvark b'daardvark
so THAT'S what my neighbour's yappy dog is doing
> generating random magic words by adding progressively more noise to "abracadabra"
You didn't *quite* end up with "Avada Kedavra", but pretty close!
until, finally, somehow, mysteriously
it turned out
that Fred Rogers was right all along:
Stop taking the above if you experience any adverse reactions...
it's the circle of life
and it moves us all
@aparrish Ablamabalabtab is so good
@aparrish the third one is just "abracadabra" though! Did the noise randomly all turn out close to zero on that one?
Server run by the main developers of the project It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!