This is a long post so let me know if any part of it (or everything!) is confusing (=
— nice/renice/process priority –
As far as nice/renice (http://www.manpagez.com/man/1/nice/) – that is the way Mac/Linux handle telling the operating system how important a process is to you when it comes to allocating resources. In windows, the equivalent would be opening up the task manager and changing the process priority from Normal -> Low.
What you're doing here is basically telling the operating system you want to run this program in the background so when it asks for resources, give it less in case some foreground program wants them. This will of course make Foldit run slower, but that may be fine.
The normal way things happen is the OS will look at all of the programs and see if they have work they want to do. Foldit always has work to do so unless other programs have something to do, the OS will say "oh sure, you can use 100% CPU Foldit". Then when you try to interact with say you're browser, it will take a non-zero amount of time for your OS to figure out that you now want your browser to have priority and to scale back what it is letting Foldit do. This is what changing the priority does. It basically makes the OS more aware that it should be harsh and stingy with the CPU when it comes to Foldit.
— Possible Bugs —
The load should be heavily centered on the CPU. There shouldn't be much disk/network activity and the RAM usage should stay fairly steady.
If there is a lot of disk or network activity because of Foldit and that is causing your system to hang, that seems like it's a bug.
If the memory usage is continually rising out of control and that's causing the hangs while you're running tools, that would be a bug as well.
If when you are running no tools and you minimize Foldit the CPU usage doesn't go down (it should go down quite significantly) that seems like it would be a bug on the Mac (I've not noticed that ever happen on Windows or Linux).
— Maxing out you CPU —
The maxing out of a CPU core while a tool is running is how it should be happening (I'll talk a little more about this below). So even when you minimize, if a wiggle is still running, it will be eating up an entire core of your CPU.
If you want to know more about the maxing out of the CPU, I'll explain further below. If this explanation is too long/rambling/doesn't make sense feel free to ask for a shorter and different explanation.
So in most user applications, you tend to display some text/graphics and wait for input such as mouse clicks/keyboard entry. These are not very intensive, and you will see they may take 1% or less of your CPU when you're not interacting with them.
In the case of most 3D video games, the situation is different. One of the differences is we have a 'game loop'. The other difference is specific to Foldit where we are running tools. I'll go over them both separately.
— Maxing out you CPU : The Game Loop —
Because you are always displaying some sort of 3D environment that can continually change, you are doing lots of computations many times a second. What Foldit does here is we set a 'desired FPS' (frames per second). This is how many times we want the screen to update per second. The higher this value the smoother everything should look when things are animating. A higher value also means here is less time to preform all those calculations.
If your CPU is fast enough, you will get that many refreshes per second and the Foldit game loop will not use any more of your CPU here. If your CPU is not fast enough, it will max out at 100% trying to update your screen as fast as it can. Most of this work is only done if Foldit is visible. When you minimize Foldit, we pause doing things like drawing and updating what the protein looks like. This is why CPU usage will drop when you minimize Foldit.
Side Note: If you are interested in the high level game loop, see: http://gamedev.stackexchange.com/questions/1589/fixed-time-step-vs-variable-time-step
What Foldit does is basically attempt to set a fixed time step, but the code is flexible enough to fall back to a variable time step if the CPU can not keep up.
— Maxing out you CPU : The Tools —
This is probably what is causing the 'excessive cpu usage' that people are seeing. When you run wiggle, you are running rosetta code (http://boinc.bakerlab.org/) in a separate thread (http://en.wikipedia.org/wiki/Thread_(computing)). This will run as fast as it can and eat up 100% of at least one core on your CPU. If it is not, then it is waiting on something slow (such as accessing memory or disk or something else). On a system with 1 or 2 cores, this can be an issue if you are trying to run other programs. The same things happens if you are running a long calculation/macro in Excel and you try to do other things. The way rosetta code works, it would not really be feasible to change the code to try and run at 1/2 speed or something like that. We trust the operating system should be able to handle giving out resources in a manner that is transparent to the user (of course this doesn't always happen as you have experienced). That's why I pointed you in the direction of the nice/renice tools. The issue is with the operating system not releasing resources to the programs you want because it doesn't know that Foldit is less important than others.