backfill: override statement_timeout to 0 in VERIFY INDEX job about cockroach

[annrpom]

Seen https://github.com/cockroachlabs/support/issues/2998

[annrpom]

As a follow-up, we should ensure other jobs kicked off by schema changes also get this override.

To tag on, we might want to audit if other session variables would land us in a similar spot.

[annrpom]

Actually, we can just make all background jobs ignore the statement timeout in here

cockroach/pkg/sql/internal.go

Lines 1131 to 1150 in b9680b4

	var sd *sessiondata.SessionData
	if ie.sessionDataStack != nil {
	// TODO(andrei): Properly clone (deep copy) ie.sessionData.
	sd = ie.sessionDataStack.Top().Clone()
	} else {
	sd = NewInternalSessionData(context.Background(), ie.s.cfg.Settings, "" /* opName */)
	}
	if globalOverride := ieMultiOverride.Get(&ie.s.cfg.Settings.SV); globalOverride != "" {
	globalOverride = strings.TrimSpace(globalOverride)
	// Prepend the "global" setting overrides to ensure that caller's
	// overrides take precedence.
	if localOverride := sessionDataOverride.MultiOverride; localOverride != "" {
	sessionDataOverride.MultiOverride = globalOverride + "," + localOverride
	} else {
	sessionDataOverride.MultiOverride = globalOverride
	}
	}
	applyInternalExecutorSessionExceptions(sd)
	applyOverrides(sessionDataOverride, sd)

ref: https://cockroachlabs.slack.com/archives/C04N0AS14CT/p1719416225444619?thread_ts=1719410054.255929&cid=C04N0AS14CT

[rafiss]

I think this diff might be what we want. We could update the comment on NewInternalSessionData to indicate this.

diff --git a/pkg/sql/internal.go b/pkg/sql/internal.go
index 24685b16219..cfb161132b4 100644
--- a/pkg/sql/internal.go
+++ b/pkg/sql/internal.go
@@ -95,6 +95,7 @@ func NewInternalSessionData(
 	sd.SearchPath = sessiondata.DefaultSearchPathForUser(username.NodeUserName())
 	sd.SequenceState = sessiondata.NewSequenceState()
 	sd.Location = time.UTC
+	sd.StmtTimeout = 0
 	return sd
 }

[rafiss]

For people who are affected by this right now, we can recommend that they configure the default statement_timeout using ALTER ROLE ALL SET statement_timeout = ... rather than the sql.defaults.statement_timeout cluster setting.

[annrpom]

i'll leave this open until i take care of my backports

[annrpom]

Re-opening as per https://cockroachlabs.slack.com/archives/C0168LW5THS/p1723150249619579