"Density Refinement Puzzle" - unstable score and undo

[LociOiling]

Normally, scores in Foldit are reasonably stable. In most cases, a saved solution opens at the same score. One exception is objectives like the Compound Library, where additional steps may be needed to recover a previous score.

Similarly , undo and redo behave in a predictable manner most of the time. (There does seem to be a bug which involves the undo graph not getting populated in some cases.)

In the Density Refinement Puzzle, the usual stability is lacking.

One problem is that the density changes from the Refine Density tool aren't recorded along with a saved solution. So a manually saved solution which is shared and opened on a separate Foldit client will load at a different score. This is a problem for evolvers and also for players who run multiple clients.

A solution for the first problem may be to simply run the Refine Density tool again after opening the shared solution. In at least some tests, this recovers the previous score. In principle, this is somewhat similar to the steps necessary to recover the Compound Library bonus.

Another problem seems to be that density changes aren't part of undo/redo. For example, I achieved a stable score of 27,398 in one client. Additional wiggling did not change the score. I then used the Refine Density tool again, and the score dropped to 27,309. At this point, wiggle loses additional points. "Restore Credit Best" restores the 27,398 score, but then wiggle drops the score again. The 27,398 score can be reached using the Undo command, but again, it's no longer stable. It seems like the Refine Density changes aren't part of what Undo undoes.

In general, there doesn't seem to be a way to get back to the original density map. Resetting the puzzle doesn't seem to do it. Restarting Foldit doesn't seem to do it. Perhaps manually deleting puzzle files could reset the density, but it's not immediately clear which file contains the updated density.

Assuming the density updates do get stored in a file someplace, there may be another issue for players who run multiple instances of Foldit from the same directory. The changes by the Refine Density tool in one client might affect the results in a different client. Most likely, this would happen when starting a new client, but the phrase "unpredictable results" might be a good description.

Also, as noted in #bugs-and-feedback on Discord, I've seen two crashes after restarting. In both cases, a second restart was successful.

[LociOiling]

After further testing, involving two freshly installed Foldit clients, I can't locate where the density information resides. So perhaps it's actually saved in the ir_solution file. The relatively small size of the ir_solution files seems to argue against that, but I can't see where else it could be.

The plan was to shake, shake sidechains, and wiggle using Auto wiggle power on both new clients. Then save the solutions and restart both clients. After restarting both clients, run Refine Density on client 1 and share the solution, opening it in client 2.

This test produced some interesting results.

Here are some screen shots which demonstrate some of the issues.

Client 2 wouldn't load the previous solution on the first two tries. Here's what it looked like after the first restart:

The previous score reached on client 2 was 26,467. Attempting to load the saved solution didn't help.

A second restart of client 2 produced the same results. A third restart was more successful, and actually loaded at a higher score, 27,123:

Client 1 didn't have a problem at restart, and similarly loaded at a higher score. After Refine Density, the score dropped, but was still 27,140:

Switching back to client 2, loading the 27,142 shared by client 1 produced still another higher score, 27,220:

(It's not clear to me how the 27,140 score on client 1 changed to 27,142 in the shared solution.)

The results for this test show that the Refine Density puzzle scoring is even more of a problem than I thought at first. I'll try another test using recipes to see if more consistent results are possible.

[rosie4loop]

From the behavior I dont think the density is stored in a file, rather it seems to be calculated based on the current structure as we start a client or open the puzzle, and update every time we click the density refinement tool.

Loading a saved solution seems to load the structure only, and score it with the previous map. To get a consistent score, everytime the structure is modified, the density must be refined.

Say we have a solution with score X. Save the solution at score X, structure X and density X, make sure density X is calculated based on structure X but not a previous solution, and this only store score X and structure X. Do some structure manipulation and density refinement to get score Y, structure Y and density Y. Loading solution X would load structure X, while keeping density Y, hence the score would based on structure X + density Y = a new score Z. To restore score X, simply refine the map on the loaded structure X.

The score of the starting structure after refinement is always the same. (Reset puzzle then do one refinement get the same score.)

[rosie4loop]

As I suggested in puzzle comments (https://fold.it/puzzles/2013729#post_77434)

Maybe in this kind of puzzles, the score should only be recorded or submitted after the density is refined on the current structure, so the score could better reflect the quality of the model built by players.

I don't know how to implement this in the game, though.

[apetrides]

@LociOiling thank you for the feedback. @rosie4loop is correct here- the current implementation of the tool works such that when the puzzle is loaded in, the density is calculated from the current state of the model. this probably explains the anomalies you were noticing with the score. the state of density is currently not saved as a part of solutions, but we can change this.

We foresaw that players might want us to save the state of density in one form or another, and this feedback has helped us confirm this so thank you!

[LociOiling]

rosie4loop wrote:

From the behavior I dont think the density is stored in a file, rather it seems to be calculated based on the current structure as we start a client or open the puzzle

Yes, that seems a good theory of how it works. It's not clear to me why the calculation keeps coming up with different results, however.

Here's what I've been seeing.

I'm just working the puzzle in the usual way. I have one version where I used Refine Density early on, and one where I avoided it.

For both versions, the solutions loaded higher when I restarted mid-game. Then after another restart, both solutions loaded lower. With some shaking, wiggling, and fuzing, solutions recovered to slightly better than their previous scores.

Starting a new track is similarly confusing. After starting a "refine1" track and using Refine Density, the score drops. At this point, Restore Credit Best restores the previous score, but then wiggle drops it back to near the lower score.

Switching back to track "default" from "refine1" likewise results in a lower score. The previous state of the "default" track is not preserved.

I expected the version with unrefined density to have a stable score like a normal puzzle. This has not been the case. Both versions show unexpected score changes when Foldit is restarted. I suspect that reloading saved solutions will show similar behavior.

[LociOiling]

(I managed to missing apetrides' answer to rosie4loop when I posted earlier, failed to hit refresh. So my reply comes off as a little uninformed.)

Saving the density state seems like the way to go. Otherwise, things like undo, changing tracks, or loading a shared solution won't work as expected.

For example, a solution shared with the group could gain points when opened, resulting in an instant evo. Usually you have to at least shake or wiggle to get the evo. Or the group-shared solution could load lower, making it difficult or impossible to evo.