Global soloist scores calculated incorrectly.

[jflat06]

Again, I understand you fall into the group of people who are willing to play every puzzle. You are looking at it from your perspective - try to think about the system as a whole, and how it will play out in practice.

Under the old system, most people will not be willing to keep up with an increased puzzle load. As they fail to keep up, they would become discouraged, and less likely to stay with the game. You might be okay with this, citing survival-of-the-fittest as the correct governing principal.

But these people are good folders, and are useful to science. The leaderboard should reflect this - NOT just who has the time and energy to tackle every puzzle.

The new system promotes a smaller number of higher quality solutions from each player, resulting in better results from puzzles for us, and also keeps people engaged in the game longer, as they are less likely to burn out. Less people burning out means more people contributing these higher quality solutions. The smaller window size allows new players to be engaged in the game sooner, increasing the likelyhood that they'll stick with it and actually MAKE it to the 4 month mark. The competition is still there, it's just focused more on who's doing well now than who's been around longer.

I'll ask you to step back for a second, and take an objective look at the design decisions between the two systems, and re-evaluate your position. Remember that just because it's the way it's always been doesn't mean it's the way it should be.

[B_2]

I think it's pandering to the trophy generation - those who won't participate unless they get immediate rewards.

I suspect you will end up with an endless parade of newbies, and your dedicated base of good experienced players will dry up. Once they (we?) leave, it's exponentially harder to get them back.

You should do everything possible to try to retain the knowledge base, not try to alienate them by screwing with the scoring.

But - perhaps the social experiment is the real science goal here, so this might be just the most recent way the scientists are poking needles at their captive white rats.

[jflat06]

This system will not alienate old players. The system remains competitive. The best folders will still occupy the top spots. If someone manages to beat you over a period of 6 weeks, dare I say it, they are a better folder than you.

I have yet to see any constructive criticism of the new system vs old, so at this point I'm sure you're just trying to get a reaction out of me, especially bringing up the topic of "captive white rats". We do not think of our players as lab rats.

The new system has very clear design advantages, as I've explained in previous posts. It is being done with the primary goal of increasing the probability of Foldit contributing to scientific discovery.

I'll ask you to please keep the discussion civil.

[B_2]

You want constructive?

1. Get rid of the ugly green bars that were just added.
2. Keep the overall 120 day puzzle window.
3. Make the scoring windows based on time, not number of puzzles

You all let the cat out of the bag last month when you revealed that we were all subjects of a psych or game theory experiment, analyzing the IRC chat, feedback and forum. How are we to know if changes thrust on us out of the blue are for the good of the science, or to satisfy the scientists studying us?

I still cannot understand how your new scoring system will help anything. Perhaps I'm just dense, or you just aren't explaining it well.

You want to run more puzzles, fine. Just start them up. Why do you have to mess with the scoring system to do that? Outside of not being able to calculate a running 120 day point total, there doesn't seem to be anything that needs to be fixed.

[jflat06]

Sorry, when I say constructive criticism, I mean that I want to understand the reasoning behind your suggestions, not just what your suggestions are. We are aware that you want a longer window, but what we're interested in is 'why'. We have very clear reasons for making it a shorter window, and I have yet to see any clear reasons not to. I've also explained the reasoning for using a puzzle-based window as opposed to a time-based window.

If you are having trouble understanding my posts, I suggest you re-read them with an open mind, as I think I've done a good job of explaining our reasoning already.

[B_2]

As I mentioned before, a longer window encourages and rewards a long term commitment to the project, which a short window does not.

A short window will reward a lucky guess or two, but staying at the top or holding position in a long-term window requires commitment and dedication the should be encouraged.

If you just want slashdot or Nature flash-in-the pan players, I guess your new scheme will achieve that, but at the expense of the real base of support and knowledge.

Obviously you place less value on the committed players than they deserve. Too bad.

[auntdeen]

There is one very good reason for a long time window for overall rankings.

It will take a while before many of us will be able to forget the months when the software was seriously screwed up. By removing the sliders and backtracking, that fixed it to some degree. A few weeks ago, diversity scoring returned, again to some degree - but unless you folks have figured out why and not shared it, it was something that was not planned for, and was not done on purpose.

My point is that the software has grown incredibly complex. Every improvement you make to it has the possibility of introducing a new (or old) bug. We all acknowledge that as "the price we pay" to play the game, and are as patient as we can be when something happens that adversely affects gameplay.

Had you had a 6 week window when the software was bad, many of the experienced players would likely have stopped playing (most of those who do handbuilding, which was extremely affected) as their rankings dropped severely. As it was, some did stop playing - and more were on the verge of quitting.

As it is, the software does keep having bugs (the ongoing crashes that bertro is having, and the fact that even on a different computer for me, mac snow leopard will hard hang during any mutate script). We also are still missing what used to be much of the fun of the game, walking.

I do understand some of the reasoning behind what I think is an attempt to increase the puzzle load without discouraging players. I like the idea of skips. I like having the beginners be able to gauge themselves to other beginners. I like the ability to see how I am doing in the different types of puzzles.

But I agree with Boots and Brick that there is no reason to severely shorten the overall rankings time period, for the reasons they have stated. I see no reason why we can't have a longer window for overall rankings, which could integrate with average number of puzzles in a new typical 4 month window (including skips).

Jflat, you were obviously assigned this job, and you've done a lot of good work. Please realize that this whole scoring thing was raised once in a dev chat with many objections on the part of the players, and what you have presented was not discussed with the players at all. Much of what you have accomplished is being received positively, but this one thing - the drastic change to the overall ranking window, is going to spark a lot of conflict since the players were unable to make their thoughts known in advance.

[B_2]

Can we go back to the original topic -

Have the 4 month solo, evolver and team leaderboards been recalculated correctly since this feedback was opened?

I think not.

Can you PLEASE at least try to get that to happen reliably? Id doesn't make any difference what the scoring window is if the leaderboards don't ever get re-calced every time a puzzle closes.

[infjamc]

I see several ways to reward a long-term commitment while still using a short window:

1. Have a parallel annual and all-time leader board (which is what I'm seeing on the "Top Soloists for Recent Puzzles" page).

2. Reduce the exponent on the score function from 7 to 5 or lower. As things now stand, one top finish is worth many above-average finishes, which could cause major fluctuations in rank for those with a high variance in performance.

3. Another possibility is to place a multiplication factor to one's score based on the amount of time that has passed since joining Foldit. For example:

+1% after 2 months
+2% after 4 months
+3% after 6 months
+4% after 9 months
+5% after 12 months (+ achievement)
+6% after 18 months
+7% after 24 months (+ achievement)
+8% after 36 months (+ achievement)
+9% after 48 months (+ achievement)
+10% after 60 months (+ achievement)

[jflat06]

From talking with Seth, my understanding is that the scoring used to be unlimited - going back to the start of the project. In response to player feedback, the window was shortened.

But the actual length of the window (4 months) was essentially arbitrary. It just 'seemed like a reasonable value'. The same applies to the new 30 puzzle value in the new system. It just 'seemed like a reasonable value'. As I've stated in the blog, these values are subject to change based on our observations and player feedback.

There had been previous player discussion about shortening the window in the thread covering the last time the scoreboards had issues: http://fold.it/portal/node/992082 (Including support from B_2 and yourself).

Again, it could be that the value is off, and it was cut too short. As you've mentioned, the issue with cutting the window too short is that it is not as tolerant to short-term issues, such as a set of weird puzzles, or a bug in the game. It also somewhat diminishes the statistical significance of the ranking. But in general, it is desirable to keep the window as short as possible while avoiding these issues.

30 puzzles / 6 weeks is a lower bound on a 'reasonable' value. My plan was to adjust this value until we reach what seems to be the optimal value. In retrospect, it may have been a better idea to start longer, and then incrementally shorten it until the optimal value was found.