A New Chapter for Foldit

Started by katfish

Susume Lv 1

The Staph aureus ED puzzle is available in devprev and main if you check show beginner puzzles, so you can compare to the same puzzle in newchapter.

Timo van der Laan Lv 1

Ran my DRW 2.3.0 minimized on both main and newchapter at the same time. Newchapter was about a factor 3 slower. Will repeat this test after the update to newchapter and investigate what is slower.

Susume Lv 1

It appears that the updates have replaced iterations based on score change with iterations based on system clock. There are some serious problems with this approach. In general, both hand folding and scripts seek to wiggle until wiggle either stops returning points or is only returning fractions of points. With clock-based iterations, it now can take 30 iterations or more (in one of my low-CI tests, 80 iterations) to reach this point of stability.

Most LUA scripts and all GUI scripts have a hard-coded number of wiggles which was judged by the author to be enough to achieve some goal, usually stability, and which will now fall far short of being enough to achieve that goal. Our existing body of scripts will cease to operate as designed. Going forward, only LUA scripts can be created that will continue wiggling until stability is reached. GUI scripts will have to have arbitrarily large numbers of wiggles coded in.

The number of iterations required to achieve stability will vary widely (200%, 300% or more) between slow and fast machines, so the same script will have very different results for different players. Even the same script on the same machine under different CPU loads will have widely differing results.

I recognize that adding a time limit to an iteration of wiggle was a response to our complaint that one iteration takes too long, but it opens a whole new can of worms, and does not address the underlying complaint, which is that wiggling to stability takes too long.

Susume Lv 1

I should have said, iterations based on some unit of work done have been replaced… Regardless, the problems created are the same.

AsDawnBreaks Lv 1

If this is kept, it would be a good idea for scripters to include a starting benchmark… Maybe a few. Can't do that with GUI though.

Susume Lv 1

There is a new thread specifically about performance (speed) issues in the feedback section: http://fold.it/portal/node/996631

For other issues, players will have an easier time finding what has been posted already, and locating new comments, if we make one thread per issue in the feedback section, starting the subject line with "Newchapter:" and choosing "Developer Preview" for the topic. This will also let us close topics where the issue has been resolved.

Rebuild/undo graph and bands in space are working correctly now; thank you!

jflat06 Staff Lv 1

Some very good points by Susume regarding the new iteration behavior.

We had actually intended to write up something about why we've made this change.

What changed? We've included a maximum internal rosetta iteration limit that prevents a single Foldit iteration from running too long. It isn't based on the system clock. The amount of 'work' done is machine-speed independent, so fast/slow computers should come up with the same answer for the same run of wiggle.

Why did we make this change? There are many implicit expectations about what a Foldit 'iteration' should be. Two of the most problematic expectations for us as developers are that 1. An iteration should take approximately x amount of time, and 2. x number of iterations should leave the protein stable.

Number 1 is continually is a problem for us because virtually any change that the Rosetta folks make to the minimizer can change the runtime of a Foldit iteration (sometimes drastically). This creates problems for script writers, who've based their scripts with assumptions about how long an iteration will take. The biggest issues are when an iteration suddenly takes 5x as long.

Likewise, the expectation that a protein will be stable after x number of iterations is also subject to changes made to Rosetta, and therefore can disrupt script functionality.

We've made these changes to the iterations in order to set things up in a way that isn't subject to changes in Rosetta. An iteration should now do some machine-independent amount of work, on a fairly fine timescale. It doesn't fulfill any expectation about stability anymore.

But stability is very useful - why would we do this? Because we want YOU to determine when a Wiggle isn't getting enough points anymore. In hand folding, that should be the same as ever - just eyeball it. In scripts, it's a little more complicated. Instead of telling Foldit to run for x iterations of wiggle, do something like this:

THRESHOLD = 0.1
STEP = 2
new_score = get_score()
do
score = new_score
Wiggle(STEP)
new_score = get_score()
while (new_score - score > THRESHOLD)

This might seem verbose, but is also much clearer. This is very similar to what Rosetta is doing underneath, and a better way of approaching things. It should give a finer degree of control over exactly how you want Wiggle to behave.

Scripts that are updated to follow this convention should be subject to much less change in Rosetta. The script should run 2 Foldit iterations of work, then check to make sure that it has gotten at least 0.1 points, and then continue or stop. The end result is that this script should always be in a very similar state of stability after exiting the loop, regardless of changes made to Rosetta.

We realize this will break quite a few scripts, which is why we've included it in the newchapter. But we think that this change should make scripts more stable in the long run.

AsDawnBreaks Lv 1

Yes, but if you do it this way, then wouldn't you have to break wiggle to check? If you do that, wiggle does not perform as well as leaving it un-broken for the same number of iterations (or at least it gains less points). Might be something good to fix, but last I knew this was still true.

spvincent Lv 1

Have the devs given any thought as to how the undo buffer would work with the above code fragment? Once a single wiggle is replaced by multiple wiggles, the undo graph is going to get cluttered with meaningless intermediate wiggle stages which are unneeded from a user standpoint. Some direct control over what goes in the undo graph will be needed by script writers I believe.