Error - segmentation fault - malloc_consolidate about steem

[mvandeberg]

We had run into a similar issue when a seed node maxes its connections. @revflash can speak more to that. We got around it by allowing the maximum number of connections to be configurable in config.ini. Try increasing the value and see if it reduces the frequency of occurrence.

[roadscape]

Thanks, I've set p2p-max-connections = 500 for now..

[roadscape]

Another segfault, afaict identical. This is running the develop branch.

3340297ms th_a       database_api.cpp:190          ~database_api_impl   ] freeing database api 124650176
3340300ms th_a       database_api.cpp:185          database_api_impl    ] creating database api 124650176
3340300ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc":"2.0","method":"get_active_votes","params":["steem-id","re-steem-id-shapeshift-claims-hack-inside-job"],"id":0}
3340300ms th_a       database_api.cpp:190          ~database_api_impl   ] freeing database api 124650176
3340301ms th_a       database_api.cpp:185          database_api_impl    ] creating database api 124650176
3340301ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc":"2.0","method":"get_accounts","params":[["berkah"]],"id":0}
3340302ms th_a       database_api.cpp:190          ~database_api_impl   ] freeing database api 124650176

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5c2e700 (LWP 6849)]
0x00007ffff6a30845 in malloc_consolidate (av=av@entry=0x7fffe8000020) at malloc.c:4165
4165    malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ffff6a30845 in malloc_consolidate (av=av@entry=0x7fffe8000020) at malloc.c:4165
#1  0x00007ffff6a31df8 in _int_malloc (av=0x7fffe8000020, bytes=20128) at malloc.c:3423
#2  0x00007ffff6a347b0 in __GI___libc_malloc (bytes=20128) at malloc.c:2891
#3  0x00007ffff72f1dad in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x0000000000b46552 in websocketpp::endpoint<websocketpp::connection<fc::http::detail::asio_with_stub_log_and_deflate>, fc::http::detail::asio_with_stub_log_and_deflate>::create_connection() ()
#5  0x0000000000b46de5 in websocketpp::server<fc::http::detail::asio_with_stub_log_and_deflate>::start_accept(std::error_code&) ()
#6  0x0000000000b470b7 in websocketpp::server<fc::http::detail::asio_with_stub_log_and_deflate>::handle_accept(std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log_and_deflate> >, std::error_code const&) ()
#7  0x0000000000ac7c55 in std::_Function_handler<void (std::error_code const&), std::_Bind<std::_Mem_fn<void (websocketpp::server<fc::http::detail::asio_with_stub_log_and_deflate>::*)(std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log_and_deflate> >, std::error_code const&)> (websocketpp::server<fc::http::detail::asio_with_stub_log_and_deflate>*, std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log_and_deflate> >, std::_Placeholder<1>)> >::_M_invoke(std::_Any_data const&, std::error_code const&) ()
#8  0x0000000000b34b9c in websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::handle_accept(std::function<void (std::error_code const&)>, boost::system::error_code const&) ()
#9  0x0000000000b4e24d in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder1<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::system::error_code> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder1<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::system::error_code>&) ()
#10 0x0000000000b4e567 in void boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>::operator()<boost::system::error_code>(boost::system::error_code const&) ()
#11 0x0000000000b4e729 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#12 0x0000000000b4e96f in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)> > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)> >&) ()
#13 0x0000000000b4ebaf in boost::asio::detail::reactive_socket_accept_op<boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::ip::tcp, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log_and_deflate::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#14 0x0000000000aba4a0 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#15 0x0000000000b5be1b in fc::asio::default_io_service_scope::default_io_service_scope()::{lambda()#1}::operator()() const ()
#16 0x0000000000c6567a in thread_proxy ()
#17 0x00007ffff7bc4182 in start_thread (arg=0x7ffff5c2e700) at pthread_create.c:312
#18 0x00007ffff6aac47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[roadscape]

Running 0.2.0...

859139ms th_a       database_api.cpp:189          database_api_impl    ] creating database api 144561904
859139ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc":"2.0","method":"get_account_count","params":[],"id":0}
859140ms th_a       database_api.cpp:194          ~database_api_impl   ] freeing database api 144561904

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff3dd8700 (LWP 26067)]
0x00007ffff65a3845 in malloc_consolidate (av=av@entry=0x7fffec000020) at malloc.c:4165
4165    malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ffff65a3845 in malloc_consolidate (av=av@entry=0x7fffec000020) at malloc.c:4165
#1  0x00007ffff65a456d in _int_free (av=0x7fffec000020, p=<optimized out>, have_lock=0) at malloc.c:4057
#2  0x0000000000aece1c in std::_Function_base::_Base_manager<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config>::*)(boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config> >, std::_Placeholder<1>, std::_Placeholder<2>)>, boost::asio::detail::is_continuation_if_running> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
#3  0x0000000000b0f725 in boost::asio::detail::reactive_socket_send_op<boost::asio::detail::consuming_buffers<boost::asio::const_buffer, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> > >, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, websocketpp::transport::asio::custom_alloc_handler<std::function<void (boost::system::error_code const&, unsigned long)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#4  0x0000000000ae3110 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#5  0x0000000000b4ba5b in fc::asio::default_io_service_scope::default_io_service_scope()::{lambda()#1}::operator()() const ()
#6  0x0000000000c786ba in thread_proxy ()
#7  0x00007ffff7bc4182 in start_thread (arg=0x7ffff3dd8700) at pthread_create.c:312
#8  0x00007ffff661f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[mvandeberg]

Looks like it happens when you close the web socket. Correct?

[roadscape]

It does appear this may happen when the connection is closed. But I'm not sure how I could tell for sure.

Note that I'm using RPC calls to the node as opposed to persistent ws connections.

[roadscape]

Received two more segfaults today whose backtraces were identical:

1205308ms th_a       database_api.cpp:195          database_api_impl    ] creating database api 156483600
1205309ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc":"2.0","method":"get_accounts","params":[["fbdan"]],"id":0}
1205310ms th_a       database_api.cpp:200          ~database_api_impl   ] freeing database api 156483600

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff3dd8700 (LWP 24810)]
0x00007ffff65a3845 in malloc_consolidate (av=av@entry=0x7fffec000020) at malloc.c:4165
4165    malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ffff65a3845 in malloc_consolidate (av=av@entry=0x7fffec000020) at malloc.c:4165
#1  0x00007ffff65a4df8 in _int_malloc (av=0x7fffec000020, bytes=20128) at malloc.c:3423
#2  0x00007ffff65a77b0 in __GI___libc_malloc (bytes=20128) at malloc.c:2891
#3  0x00007ffff6e64dad in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x0000000000b5a404 in websocketpp::endpoint<websocketpp::connection<fc::http::detail::asio_with_stub_log>, fc::http::detail::asio_with_stub_log>::create_connection() ()
#5  0x0000000000b5ac75 in websocketpp::server<fc::http::detail::asio_with_stub_log>::start_accept(std::error_code&) ()
#6  0x0000000000b5aecf in websocketpp::server<fc::http::detail::asio_with_stub_log>::handle_accept(std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log> >, std::error_code const&) ()
#7  0x0000000000b17aa5 in std::_Function_handler<void (std::error_code const&), std::_Bind<std::_Mem_fn<void (websocketpp::server<fc::http::detail::asio_with_stub_log>::*)(std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log> >, std::error_code const&)> (websocketpp::server<fc::http::detail::asio_with_stub_log>*, std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log> >, std::_Placeholder<1>)> >::_M_invoke(std::_Any_data const&, std::error_code const&) ()
#8  0x0000000000b1d2cc in websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>::handle_accept(std::function<void (std::error_code const&)>, boost::system::error_code const&) ()
#9  0x0000000000b63add in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder1<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::system::error_code> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder1<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::system::error_code>&) ()
#10 0x0000000000b63db3 in void boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>::operator()<boost::system::error_code>(boost::system::error_code const&) ()
#11 0x0000000000b63f79 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&)> (websocketpp::transport::asio::endpoint<fc::http::detail::asio_with_stub_log::transport_config>*, std::function<void (std::error_code const&)>, std::_Placeholder<1>)> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()

[roadscape]

0.4.2:

5367ms th_a       database_api.cpp:200          ~database_api_impl   ] freeing database api 191130176
5401ms th_a       database_api.cpp:195          database_api_impl    ] creating database api 191130176
5401ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc":"2.0","method":"get_dynamic_global_properties","params":[],"id":0}

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff3dd8700 (LWP 27010)]
0x00007ffff65a3845 in malloc_consolidate (av=av@entry=0x7ffff68e3760 <main_arena>) at malloc.c:4165
4165    malloc.c: No such file or directory.
(gdb) bt
#0  0x00007ffff65a3845 in malloc_consolidate (av=av@entry=0x7ffff68e3760 <main_arena>) at malloc.c:4165
#1  0x00007ffff65a456d in _int_free (av=0x7ffff68e3760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:4057
#2  0x0000000000ba943a in websocketpp::http::parser::response::~response() ()
#3  0x0000000000bcfcc0 in websocketpp::connection<fc::http::detail::asio_with_stub_log>::~connection() ()
#4  0x0000000000818129 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#5  0x0000000000b79358 in std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (websocketpp::connection<fc::http::detail::asio_with_stub_log>::*)(websocketpp::connection<fc::http::detail::asio_with_stub_log>::terminate_status, std::error_code const&)> (std::shared_ptr<websocketpp::connection<fc::http::detail::asio_with_stub_log> >, websocketpp::connection<fc::http::detail::asio_with_stub_log>::terminate_status, std::_Placeholder<1>)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
#6  0x0000000000b7f7c3 in std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, std::error_code const&)> (std::shared_ptr<websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config> >, std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, std::_Placeholder<1>)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) ()
#7  0x0000000000bc7ad3 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, boost::system::error_code const&)> (std::shared_ptr<websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config> >, std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, boost::system::error_code const&)> (std::shared_ptr<websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config> >, std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, std::_Placeholder<1>)> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#8  0x0000000000bc7cbf in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder1<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, boost::system::error_code const&)> (std::shared_ptr<websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config> >, std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock>, boost::asio::waitable_timer_service<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock> > > >, std::function<void (std::error_code const&)>, std::_Placeholder<1>)>, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<fc::http::detail::asio_with_stub_log::transport_config>::*)(std::shared_ptr<boost::asio::basic_waitable_timer<std::chrono::_V2::steady_clock, boost::asio::wait_traits<std::chrono::_V2::steady_clock
[....]

And another:

2770641ms th_a       database_api.cpp:200          ~database_api_impl   ] freeing database api 188000512
2770745ms asio       asio.cpp:118                  operator()           ] Caught unhandled exception in asio service loop: bad_function_call
2770746ms th_a       database_api.cpp:195          database_api_impl    ] creating database api 188000512

Program received signal SIGSEGV, Segmentation fault.
malloc_consolidate (av=av@entry=0x7ffff68e3760 <main_arena>) at malloc.c:4151
4151    malloc.c: No such file or directory.
(gdb) bt
#0  malloc_consolidate (av=av@entry=0x7ffff68e3760 <main_arena>) at malloc.c:4151
#1  0x00007ffff65a4df8 in _int_malloc (av=0x7ffff68e3760 <main_arena>, bytes=1608) at malloc.c:3423
#2  0x00007ffff65a77b0 in __GI___libc_malloc (bytes=1608) at malloc.c:2891
#3  0x00007ffff6e64dad in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x000000000084da5e in fc::api<steemit::app::database_api, fc::identity_member>::api<std::shared_ptr<steemit::app::database_api> >(std::shared_ptr<steemit::app::database_api> const&) ()
#5  0x000000000084dfc7 in std::_Function_handler<std::shared_ptr<fc::api_base> (), void steemit::app::application::register_api_factory<steemit::app::database_api>(std::string const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000832dc1 in std::function<std::shared_ptr<fc::api_base> ()>::operator()() const ()
#7  0x0000000000837895 in steemit::app::detail::application_impl::create_api_by_name(std::string const&) ()
#8  0x0000000000843342 in steemit::app::detail::application_impl::reset_websocket_server()::{lambda(std::shared_ptr<fc::http::websocket_connection> const&)#1}::operator()(std::shared_ptr<fc::http::websocket_connection> const&) const ()
#9  0x0000000000ba288b in fc::http::detail::websocket_server_impl::websocket_server_impl()::{lambda(std::weak_ptr<void>)#3}::operator()(std::weak_ptr<void>) const::{lambda()#1}::operator()() const ()
#10 0x0000000000ba3139 in fc::detail::void_functor_run<fc::http::detail::websocket_server_impl::websocket_server_impl()::{lambda(std::weak_ptr<void>)#3}::operator()(std::weak_ptr<void>) const::{lambda()#1}>::run(void*, fc::http::detail::websocket_server_impl::websocket_server_impl()::{lambda(std::weak_ptr<void>)#3}::operator()(std::weak_ptr<void>) const::{lambda()#1}) ()
#11 0x0000000000ab3e53 in fc::task_base::run_impl() ()
#12 0x0000000000ab1e2e in fc::thread_d::process_tasks() ()
#13 0x0000000000ab20b6 in fc::thread_d::start_process_tasks(long) ()
#14 0x0000000000d5d731 in make_fcontext ()
#15 0x0000000000000000 in ?? ()

[roadscape]

FTR: Zero crashing bugs in the last 8 days, using build from develop branch @ 4ef97be

This includes:
#10 - segmentation fault - malloc_consolidate
#11 - double free or corruption
#26 - corrupted double-linked list
#43 - segmentation fault - __memcpy_sse2_unaligned
#48 - segmentation fault - _int_malloc

[roadscape]

Haven't seen this error (or any of the others referenced) since my last update... closing.