Katfish talks newchapter!

Started by katfish

katfish Lv 1

After a lively discussion in veteran chat yesterday during which I promised to write a blog entry about newchapter, here I am delivering the goods! This post will approach newchapter through the lens of why we play Foldit in the first place – to aid scientific research and discovery. I naturally coerced asked our very own bkoep to help kick off my post with a very simple explanation of a key question: What makes newchapter so critical to the scientists working on Foldit? Here’s what he had to say:

One of the things that makes Foldit a good scientific tool is that we can make direct comparisons against Rosetta. The idea is that Rosetta is equivalent to cold Foldit (without the players). When this is the case, then every difference between a Rosetta result and a Foldit result can be attributed to the players. On the other hand, if there are discrepancies (in the score function, for example) between Rosetta and cold Foldit, then we can't know whether the different results are attributable to the players or to the discrepancies. For example, with ED puzzles 813/816 Rosetta was able to find a better match to the density than Foldit players—presumably because Rosetta was able to sample 100,000s of models fairly quickly, and there were no identifying features in the density for Foldit players to work with. But because Rosetta was using the latest score function, we can't be sure that the score function wasn't the missing factor—maybe the newchapter changes to density scoring would have directed Foldit players to the correct structure.

As you may have deduced, the changes in newchapter are already incorporated in Rosetta and introduce significant improvements to scientific accuracy. Turns out these changes are so critical that asking you to work on our upcoming set of shiny new science puzzles without newchapter would yield scientifically inferior results and be a gross mishandling of your valuable time and effort.

That said, are we anxious to push newchapter to main so we can put brand new puzzles into your hands? YESS!!! Can we wait to do this until you’ve had a chance to modify scripts and establish new procedures for helping newbies? For a little while… (but do reserve the right to pout about it ;p). So, let’s compromise by pushing back the release date to allow for advanced preparations, the new official release to main being on Monday January 27th.

However, let’s not forget about CASP which is coming up in just four short months. The unapologetically competitive side of me hesitates to push this release too far into the future. The sooner we can start to tackle the transition, the more time we’ll have to prepare and optimize for CASP 2014. If we can pull this off and take full advantage of the scientific accuracies newchapter improves, we will surely kick butt.

As I bring this post to an end, I can only assume that it has likely only functioned to inspire 10, maybe 20 thousand more questions about newchapter that I have no hope of answering on my own. And so I will defer to the experts by declaring…a…SCIENTIST CHAT! Let’s get together to talk and about how we can use this change to our advantage as gamers, but also to make some incredible scientific advancements as well.

We will see you at 11:00am Pacific on Wednesday the 22nd (check your local time here!) for the Scientist Chat. Please post your scientific questions in the comments so we can be sure to cover it during the event!

  • katfish

brow42 Lv 1

Did your comparisons of players' solutions and methods for prediction or other puzzle types in Foldit, versus R@H, lead to changes in Rosetta?

auntdeen Lv 1

For the consideration that you, Seth, and the other devs are showing for the players.

And thanks for arranging a scientist chat.

We are all excited about this, and are happy that there will be more time for bug hunting (and fixes) so that the launch will be successful!

AsDawnBreaks Lv 1

A couple tings I would like brought up, as I won't be able to be there.

1) Jflat made this comment:

"But stability is very useful - why would we do this? Because we want YOU to determine when a Wiggle isn't getting enough points anymore. In hand folding, that should be the same as ever - just eyeball it. In scripts, it's a little more complicated. Instead of telling Foldit to run for x iterations of wiggle, do something like this:

THRESHOLD = 0.1
STEP = 2
new_score = get_score()
do
score = new_score
Wiggle(STEP)
new_score = get_score()
while (new_score - score > THRESHOLD)

This might seem verbose, but is also much clearer. This is very similar to what Rosetta is doing underneath, and a better way of approaching things. It should give a finer degree of control over exactly how you want Wiggle to behave.

Scripts that are updated to follow this convention should be subject to much less change in Rosetta. The script should run 2 Foldit iterations of work, then check to make sure that it has gotten at least 0.1 points, and then continue or stop. The end result is that this script should always be in a very similar state of stability after exiting the loop, regardless of changes made to Rosetta."

What has been experienced is that if you wiggle for one iteration, stop, then wiggle for another iteration, it will gain less than if you wiggle for two iterations at one time. It seems that if we went with the scheme Jflat was suggesting, it would make scripts much less efficient. Has this been thought of? Has the issue been fixed, or could it be? And last I checked, this would be IMPOSSIBLE in the GUI interface, so that renders GUI basically useless.

2) Also, to prevent old scripts from breaking, could there perhaps be a "new" wiggle function using the new iteration mechanism, and leave the old one working so older scripts are still usable? This would somehow have to be implemented into GUI as well.

3) In general, why is the behavior that there is gain over many iterations (50, even a hundred or two) gaining a few points in a much longer time span than previously, where everything would gain within the first couple iterations, and point gains were usually in large chunks?

Thoughts on these matters from the devs would be appreciated. Please let me know if I need to clarify, I would be more than happy to do so.

spmm Lv 1

Sounds I haven't had a chance to try newchapter as yet. Will it be introduced to devprev first and then main or both at the same time?

bkoep Staff Lv 1

Due to some significant changes in the new score function, we don't want newchapter clients competing in main puzzles. You can try out newchapter by switching your client's update group to "newchapter" (see http://fold.it/portal/node/996588), although you may only be able to play the newchapter puzzles.

The newchapter update will be pushed to main and devprev at the same time, scheduled for Monday, January 27.

spmm Lv 1

Thanks bkoep
There has been a heat wave here and sitting in the server aisle in the spare room is quite unpleasant, hence my limited play time.
Seems to me that the new Wiggle Power settings are a compromise between playability and 'have you heard of Rosetta at home?"

So if the science still feels that it needs some humans and is prepared to let us have the flexibility of arriving at our own conclusions (fun) then lets just get on with it.

Yes some scripts don't work but those are not necessarily shared across the community anyway. A high ranking player has mentioned 'thousands of scripts' which will need to be changed.
So share the load and open up the script pool :D.

Not really interested in wasting energy resources.

viosca Lv 1

Not sure if this is the correct place for this.
It would be interesting if wiggle took a callback function, the returned value which would signal if it should stop.

Also, if integrator and derivatives of the score values were provided, then PID controllers could be created in the callback routines to do things like control the rate of change of CI.

-Randy