New proposal for social structure of foldit, in a more general, abstract form

[Ignacio]

Auntdeen and you made convincing statements, and I agree. I stressed too much the importance of the "top score" in my post, and your points are correct. However, I also suggested the top 10 solutions from the soloists to be made available for everybody. If soloists work fully independently, as I suggested, it is likely that al these solutions are quite different. However, after reading all that you too said, I think 20 or 30 could be even better.

I did not think about the evolving score, but you are right. A simple option would be to give the evolvers as many points as the difference between the score when they first get the solution and the score when they end. On one hand this seems quite strange, given that it would mostly benefit those evolvers that dramatically improve bad solutions. However, a nice idea also emerges: now evolvers work is very technical, being concentrated on improving a bit the best solution of a group. With the other score, evolvers would be interested in improving as much as possible any solution, no matter how bizarre. Some of the solutions could perhaps make it to the top scores by that divergent, creative work, even if the soloists that first built them failed.

So the problem is just avoiding people start evolving solutions with a -900000 value and winning the competition with a single wiggle. Although this would be in general difficult if only the top 20-30 scores or so can be evolved, it still could occur, especially at the beginning of a puzzle. The simplest solution probably is allowing evolvers to start only some hours after the beginning of the puzzle, so the top soloist scores are already quite good. After all, my proposal goes in the direction of favoring that all people tries first some soloist work and later, if that fails or become tired, shifts to evolving. This could be a way to favor that.

A further refinement derives from another of my suggestions, which was to make all people become evolvers some hours before the end of the puzzle. An additional possibility would be to have a first classification for "early evolvers" for those that improve solutions before that point and a second classification for "last-minute evolvers" in which everybody would be working with the already highly evolved solutions for further improvement. Again, in both cases, the value would be the difference between the scores before and after the evolver works with the solution. Still, now I think that, if this two classifications existed, my idea of forcing everybody to become evolver becomes just too rigid, it is totally unnecessary. It would be simpler to open the "last-minute evolver" competition and leave people choose between continuing soloist work and starting last minute evolver work in those say last 12 or 24 hours of the puzzle.

I am certain I can come up with some other ideas for the evolver scores tomorrow, but now it is time for me to sleep. Thanks for your interesting feedback.

Ignacio

[infjamc]

Re: Ignacio

Overall, I have to agree to disagree with your suggestions for the reasons that others have already stated above. But just to play devil's advocate, here are a few ideas that I see as a possible compromise:

1. Instead of banning non-public recipes altogether, why not implement a small penalty for running them (say, it would cost you 1 point per 100 lines of code executed from a private script)?

2. Instead of scrapping groups for competitive purposes, here's another idea:

• Allow people to share solutions not only with yourself or your group, but also with everyone;
• Revamp the scoring system to incentivize sharing solutions with the rest of the community. For example, suppose that Player A uploads a solution to the public repository and Player B manages to evolve it, both A and B will receive a bonus to their score.

3. There's another way to discourage "efficiently copy[ing] other people" or "evolv[ing] the best solution of a group": Awarding points only to the top X players/teams on certain puzzles. That way, one cannot simply count on copying a teammate's solution for a score that's "high enough." Competition would also be fiercer, which might encourage people to experiment with radical explorations if they know that the only way they can get on the scoreboard is via a "Hail Mary rebuild"

[tokens]

Regarding design puzzles:

I don't think AD is good in design puzzles because we literally copy each other. However I myself have generated some good designs for the freestyle design puzzles, which other people in my group have learned from and made similar solutions. Conversely I have learned about which approaches worked by looking at other peoples solutions from my group. This is not copying, this is learning from other peoples experiences, and I would love if we could share the information created in our group with everyone in foldit. Again this points to the idea of letting everyone explore the top solutions after a puzzle has been closed.

My main point is this: Being in a group like AD gives an advantage because you learn from other peoples experiences, not because you copy the best scoring solution.

[auntdeen]

Sorry, your summation of CASP submissions is incorrect: "it must be exceptional that a very poor scorer has 1) a peculiar fold (otherwise you would choose a better scorer) and 2) a good energy score".

I am extremely familiar with the diversity scores from my team, which is a large sampling of foldit in general, because to date I have done 175 CASP roll & CASP 10 individual submissions for my team. The Contenders, Void Crushers and AD took beta_helix up on his offer in this post back in the winter: http://fold.it/portal/node/991099. Each of the participating teams is sent only their team's scores, and runs them through Pymol to see the diversity (Susume is the wonderful volunteer in our group who does this). Beta_helix sends to the 3 teams his picks for each CASP target - he has first choice, of course - so that we do not duplicate any of his submissions.

So yes, I know which puzzles Foldit chooses for submissions (and can look up ranks), and I know for my own team. And I can tell you that your assumptions are completely wrong. It is a very rare puzzle, in fact, that has submissions from Foldit that are all from the top ranks.

What Foldit needs for scientific purposes is diversity. That's why many were upset when the software was not rewarding handbuilding for a period of time - that's where the diversity comes from, not scripts.

In fact, Foldit at the moment is benefiting greatly from having the team structure that it has - instead of 5 submissions per CASP target, we have 20 - and more diversity that way in the submissions.

Your perception is yours - I see it completely differently. I'd rather focus instead on the teaching of handwork with better tools than we now have - webinars, a place to post HD videos and some standard free software to make videos that we could make use of for teaching. I'd rather the devs be able to come up with something that rewards diversity (and yes, it's my understanding that they are working on that).

There is an old post in the forum ( http://fold.it/portal/node/266290 ) that is a lively discussion back in 2008 about groups and scoring. In that thread, Zoran had this to say: "our goal is both to provide the engaging game play experience, and to eventually enable everyone to advance science. Groups serve a purpose for both goals. We suspect that there a number of skills involved in the protein folding process, and groups would enable us to discover much better solutions by teaming up folks that are very good at finding initial good scores, and folks that have a knack for improving such solutions further. This is why the groups are important."

…And because Susume and I look at all of our team's diversity scores for each CASP puzzle, and compare the pictures in Pymol, I can also tell you that you have another incorrect assumption "What is decreasing diversity is people massively copying other people solutions, as happens now"… What happens in our group, which is all that I can speak to (but suspect it is the same for the other top teams), is that it is extremely unusual for us to see that someone in our group has loaded a high solution and and used it as a guide.

One last note - you have commented here that you were "stunned by the good ideas developed by your group when I saw them for the first time. It was not the recipes, but the IDEAS that were novel when compared with what it was available". Two things, imho - all the top groups have novel ideas… but they are born from the easy give and take of a close knit chat (group) room. Someone tries something and says - hey, I just tried something here, can this be scripted? Someone else chimes in to say well, that could be limiting, why not do it this way instead? Another person adds an enhancement…

How would you expect that easy flow of ideas for scriptwriters to draw upon without groups?

[marie_s]

I think , people have the right to have secret strategies, recipes and so on, to be in a team or not, to share solution in the middle or the game of not, to compete by teams… an it is freedom.
All your propositions are resctrictions of freedom.
Like many limitation of freedom, they are difficult to implement, you think to have a police force to restrict what a player say to another player in the soloist part?

Sharing our solution at each step of a puzzle with everybody is not efficient for the purpose of the game and only because of this, we are ask not to do it.

we have not the same way to participate :

• many players are not in a team like me but I dont see why
• some play all puzzles, some choose some puzzles to play
• some like design puzzles and do them first,
• some mainly evolve only and dont care about solist score,
• some share their results, many dont
• some spoke, some dont,
• some ask advice, some dont.

I think I am better at design and denovo puzzles that on refinment, and I have statistics to proove it, so the most efficient way I have to participate is to do only design puzzles. I dont, because, you know, it is a game not a work so I play the puzzles I like the way i like even if the score show that I am in a bad road.

I dont think you have the knowledge on how I and many other folders play this game.

[Ignacio]

Thanks a lot for your post. My perception has indeed changed. I still think I am right in several points. One, make soloists work alone, make their evaluations truly independent, precisely to encourage diversity. Collaboration among players should start in a different, evolver phase. This indeed would be a deeep change in the structure that we have now. Two, make all tools open, to allow all players to compete in equal conditions. Another significant social change. Three, encourage scriptwriters by giving them a formal evaluation.

But, with your information, I think I was mistaken when I proposed eliminating or otherwise modyfing the structure of groups. I see now groups may encourage diversity more than they damage it. If indeed groups get totally different significant solutions very often (something that I did not think it was happening), then they are clearly another good way to increase diversity.

Additional ideas: good scriptwriters should be encouraged to collaborate with people in different groups to get more ideas, more programs, and also more points in their ranking. Once you eliminate group-only recipes, that should be easy, because scriptwriters would work for the whole community.

Another idea is that perhaps is bad to save only the top solutons for each player. It is not unusual to find two very different structures with quite similar scores. A typical case is when locating an external helix in two different places gives you almost identical scores. For CASP especially, would it be possible to encourage players to send a second solution (not to be awarded points) if it is within a given range of points of their first one and the structures are truly different?.

A final idea is that the current evaluation tools could probably be improved by giving additional value to the players that come up with peculiar structures, give points for creating new folds. I don't know how feasible would be that.

In a final, more personal aside, if the score system is so different from the true significance of a solution as to be statistically equally good to be first or 12th, I think playing becomes quite less attractive. Especially, working hard to "perfect" a solution, taking hours to increase the last few points, barely makes sense. Also, getting a score as evolver by improving a solution a couple of points may be perfectly absurd.

Thanks again

Ignacio

[AsDawnBreaks]

My thoughts exactly. I do like the download number.

[infjamc]

DISCLAIMER

To be absolutely clear, personally I would NOT like to see #1 or #3 actually being implemented because the cure might be worse than the disease. (For example, #3 is especially bad because it could have the side effect of causing people to stop playing if they feel that they cannot be competitive enough to earn points.) Again, I'm just brainstorming out loud for the sake of discussion.

[infjamc]

Getting a score as evolver by improving a solution a couple of points may be perfectly absurd.

For this issue, a simple fix would be handing out evolver credit not by the top score, but by the number of points gained. To prevent people from abusing the system by uploading the starting configuration and letting a teammate wiggle it, the scores can be weighted. For example, the score gain must occur above the median score would count, and a higher weight could be applied for the score gains closer to the top score (a quadratic function would suffice).

[infjamc]

Oops, there was a typo. The start of the third sentence should read "the score gain must occur above the median score for it to be counted…"

Anyway, to clarify the idea, here's an example. Suppose that the median score is 10000, the top score is 12000, and the function is (end_score_above_median^2 - start_score_above_median^2)/40000.

Evolving from 10000 to 12000: 100 points
Evolving from 9000 to 11000: 25 points (remember, only the portion above 10000 would count)
Evolving from 11000 to 12000: 75 points
Evolving from 11990 to 12000: 0.9975 points
Evolving from 11900 to 12000: 9.75 points