mystery crashes in "Puzzle::intro_puzzle_data" after long-running recipes

[LociOiling]

Just received two similar crashes after running two different recipes for a long time. Neither crash generated a debug.txt, just an unhandled exception in log.txt.

I'll attach the complete logs in a follow-up.

From client 1:

UNHANDLED EXCEPTION
  1: standalone::application::boinc::Puzzle::intro_puzzle_data +52 bytes (no line)
  2: standalone::application::StandaloneApplication::show_next_text_bubble +79 bytes (no line)
  3: boost::function0::operator() +120 bytes (no line)
  4: interactive::gui::Button::generate_click +85 bytes (no line)
  5: boost::function0::operator() +120 bytes (no line)
  6: interactive::application::shared::HotkeyTable::execute +79 bytes (no line)
  7: standalone::application::StandaloneApplication::on_key_down +304 bytes (no line)
  8: boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,boost::_mfi::mf1<void,standalone::application::StandaloneApplication,enum interactive::input::Key>,boost::_bi::list2<boost::_bi::value<standalone::application::StandaloneApplication *>,boost::arg<1> > >,void,enum interactive::input::Key>::invoke +19 bytes (no line)
  9: boost::function1<void,enum interactive::input::Key>::operator() +123 bytes (no line)
 10: interactive::gui::EventHandler::on_key_down +90 bytes (no line)
 11: interactive::gui::TopLevel::on_key_down +204 bytes (no line)
 12: interactive::guiwrapper::GUIWrapper::generic_key_down_func +79 bytes (no line)
 13: boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,boost::_mfi::mf1<void,interactive::guiwrapper::GUIWrapper,enum interactive::input::Key>,boost::_bi::list2<boost::_bi::value<interactive::guiwrapper::win32::Win32Wrapper *>,boost::_bi::value<enum interactive::input::Key> > >,void>::invoke +16 bytes (no line)
 14: boost::function0::operator() +120 bytes (no line)
 15: interactive::guiwrapper::GUIWrapper::enqueue_event +76 bytes (no line)
 16: interactive::guiwrapper::win32::Win32Wrapper::win_key_func +576 bytes (no line)
 17: interactive::guiwrapper::win32::Win32Wrapper::WndProc +567 bytes (no line)
 18: AddClipboardFormatListener +1179 bytes (no line)
 19: DispatchMessageW +2426 bytes (no line)
 20: CallWindowProcW +141 bytes (no line)
 21: glDebugEntry +3687 bytes (no line)
 22: AddClipboardFormatListener +1179 bytes (no line)
 23: DispatchMessageW +2426 bytes (no line)
 24: DispatchMessageW +558 bytes (no line)
 25: DispatchMessageA +16 bytes (no line)
 26: interactive::guiwrapper::win32::Win32Wrapper::start +389 bytes (no line)
 27: interactive::application::interactive_main +2117 bytes (no line)
 28: game::application::game_main +1321 bytes (no line)
 29: library_main +21 bytes (no line)
 30: no symbol (no line)
 31: no symbol (no line)
 32: no symbol (no line)
 33: BaseThreadInitThunk +36 bytes (no line)
 34: RtlValidSecurityDescriptor +282 bytes (no line)
 35: RtlValidSecurityDescriptor +234 bytes (no line)
</pre>

From client 4:
UNHANDLED EXCEPTION
  1: standalone::application::boinc::Puzzle::intro_puzzle_data +52 bytes (no line)
  2: standalone::application::StandaloneApplication::show_next_text_bubble +79 bytes (no line)
  3: boost::function0::operator() +120 bytes (no line)
  4: interactive::gui::Button::generate_click +85 bytes (no line)
  5: boost::function0::operator() +120 bytes (no line)
  6: interactive::application::shared::HotkeyTable::execute +79 bytes (no line)
  7: standalone::application::StandaloneApplication::on_key_down +304 bytes (no line)
  8: boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,boost::_mfi::mf1<void,standalone::application::StandaloneApplication,enum interactive::input::Key>,boost::_bi::list2<boost::_bi::value<standalone::application::StandaloneApplication *>,boost::arg<1> > >,void,enum interactive::input::Key>::invoke +19 bytes (no line)
  9: boost::function1<void,enum interactive::input::Key>::operator() +123 bytes (no line)
 10: interactive::gui::EventHandler::on_key_down +90 bytes (no line)
 11: interactive::gui::TopLevel::on_key_down +204 bytes (no line)
 12: interactive::guiwrapper::GUIWrapper::generic_key_down_func +79 bytes (no line)
 13: boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void,boost::_mfi::mf1<void,interactive::guiwrapper::GUIWrapper,enum interactive::input::Key>,boost::_bi::list2<boost::_bi::value<interactive::guiwrapper::win32::Win32Wrapper *>,boost::_bi::value<enum interactive::input::Key> > >,void>::invoke +16 bytes (no line)
 14: boost::function0::operator() +120 bytes (no line)
 15: interactive::guiwrapper::GUIWrapper::enqueue_event +76 bytes (no line)
 16: interactive::guiwrapper::win32::Win32Wrapper::win_key_func +576 bytes (no line)
 17: interactive::guiwrapper::win32::Win32Wrapper::WndProc +567 bytes (no line)
 18: AddClipboardFormatListener +1179 bytes (no line)
 19: DispatchMessageW +2426 bytes (no line)
 20: CallWindowProcW +141 bytes (no line)
 21: glDebugEntry +3687 bytes (no line)
 22: AddClipboardFormatListener +1179 bytes (no line)
 23: DispatchMessageW +2426 bytes (no line)
 24: DispatchMessageW +558 bytes (no line)
 25: DispatchMessageA +16 bytes (no line)
 26: interactive::guiwrapper::win32::Win32Wrapper::start +389 bytes (no line)
 27: interactive::application::interactive_main +2117 bytes (no line)
 28: game::application::game_main +1321 bytes (no line)
 29: library_main +21 bytes (no line)
 30: no symbol (no line)
 31: no symbol (no line)
 32: no symbol (no line)
 33: BaseThreadInitThunk +36 bytes (no line)
 34: RtlValidSecurityDescriptor +282 bytes (no line)
 35: RtlValidSecurityDescriptor +234 bytes (no line)
</pre>

[LociOiling]

Logs attached. The client 1 log shows evidence of opening a second client in the same directory to share a solution. This move seems to cause lots of nulls (0x00s) in the middle of the log.

No evidence of nulls in the client 4 log.

Client 1 was running a mutate recipe, client 4 was running a local wiggle recipe. The client 1 recipe ran to completion, while the client 4 recipe was cancelled. Both crashes occurred trying to a select all followed by auto SS. I'm not sure exactly where things went wrong. I probably used the home key to center the protein before trying the select all. At some point, the clients just stopped responding.

[jflat06]

Weird… and this wasn't on an intro puzzle?

[LociOiling]

No, these were regular puzzles, I think client 1 was on 1546 and client 4 was on 1545.

It might be helpful to include the puzzle name or id at some point in the log file. I'd also go for timestamp, process id, and thread id on log messages. We could probably do without "playing sound: 4" and "delta score: 0" to compensate.

[joshmiller]

Just following up on this. I think we have solved the issue - a new patch will go to devprev soon and we will keep an eye out to see if this happens again.