(explained) x86: neural net mutate gives different sequence when repeated from same starting position

Started by Bletchley Park

Bletchley Park Lv 1

I would expect the neual net mutate to produce the same sequence from the same starting position every time (consistency). Instead it produces different combinations each time it is run. I lost trust in this algorithm.

BUG

bkoep Staff Lv 1

There is some randomness built into the underlying algorithm (the same is also true for classic Mutate). We mentioned this briefly in our blog post about Neural Net Mutate, but we did not get into the details.

Under the hood, the algorithm actually predicts probabilities for each of the 20 amino acid (AA) types at each position. When it iterates over all the different positions, it picks an AA at each position based on the probabilities. There are two sources of randomness:

  1. The order in which it iterates over the different positions. The algorithm doesn't have to go from front-to-back; it can jump around to and mutate different positions in different order. This influences the probabilities of each position. The same thing is true for classic Mutate.
  2. The algorithm doesn't necessarily pick the highest probability AA at each position. While benchmarking this algorithm, we've found that we can give the algorithm some leniency here. If we let the algorithm pick other high-probability AAs, it still creates successful designs, but this leniency also yields increased diversity in the designs. The same thing is true for classic Mutate, except that it generally creates designs with a lower success rate.

Bletchley Park Lv 1

Thank you for the more detailed explanation. I could imagine that with reason 1 there could be a mode where the network evaluates all options and chooses the best scoring one.
Would it be possible to implement argument options in LUA for 1 and 2 ? I can imagine that we then have more control over the functionality and repeatability when scripting.

bkoep Staff Lv 1

I could imagine that with reason 1 there could be a mode where the network evaluates all options and chooses the best scoring one.

Yes this is possible in theory, but this is essentially the same as classic Mutate (optimizing based on Foldit score).

Would it be possible to implement argument options in LUA for 1 and 2 ? I can imagine that we then have more control over the functionality and repeatability when scripting.

It is possible, and it's something we may consider in the future.