Users should have feedback and be aware when network-related tasks fail. Otherwise, they'll be left with a blank, motionless screen without a clear indication of what happened and what to do next.
Consider adding an option to retry as well (e.g., a retry button).
Notes
Using try/catch to provide error alerts when exceptions occur.
StreamBuilder and FutureBuilder have the snapshot object that provides the connection state (success, failure, error, etc.). Use this to achieve conditional rendering.
With Twitch's implementation of IRC, there are various tag and commands that are sent along with each IRC chat message. These tags contain various key-value pairs related to the message and command (e.g., name color, badges, emotes used, subscription status, etc.). These tags need to be handled accordingly and should influence the rendering of messages if necessary.