Availability of the Splunk Service
Our Splunk environment is a hybrid cloud solution – with the primary “Splunk Web” user experience (and the primary indexing tier) delivered by Splunk’s “Splunk Cloud” SaaS offering. As Splunk Cloud matures, we expect fewer disruptions to the user experience. For the time being, some interruptions are expected (during Maintenance Windows) on a regular basis. We have worked with Splunk to identify preferred “windows” of time for maintenance both to minimize the likelihood that disruptions will impact users, but also to help us all manage expectations around those disruptions.
In short, our preferred windows are for 9pm – 6am daily, and any time over the weekend – with a preference for changes with greater chance for impact on the service and its users to occur on Wednesday mornings. See our guidance to Splunk for more details.
Disruptions While Actively Using Splunk Web User Interface
Splunk has evolved to offer a type of clustered “rolling” restart that minimizes disruption. But it isn’t yet seamless. If you happen to be using the service when the Splunk Cloud cluster is being restarted (i.e., during a maintenance window), your edits to an unsaved object (such as a report or dashboard) may be lost, your ad hoc query may be canceled.
If you ever experience this type of disruption, you should be able to resume work immediately by refreshing your page – again, you may lose any unsaved edits, but you will be redirected to another cluster member that has already restarted, and you should be able to resume work immediately.
One can minimize the risk of this (and its negative consequences) by saving often and avoiding the use of Splunk during times identified as candidates for Maintenance Windows.
Scheduling Searches (Scheduled Reports / Alerts / Summary Indexing)
Scheduled searches may be impacted (delayed or canceled) by disruptions to service – but there are things you can do to reduce the likelihood of impact.
- It is, generally speaking, safe to schedule searches during times which are identified as candidates for maintenance windows. If you want to reduce the likelihood that your scheduled search will be impacted by a maintenance activity, however, you may wish, as your use case allows, to schedule your searches outside of the maintenance window schedule guidance. Reference the maintenance window guidance.
- Be sure to set your scheduled search “window” to an appropriate time for your use case – preferably greater than 5 minutes. The idea here is to set a window that is best for your use case, but large enough to remain scheduled (queued) while the cluster member restarts (~5 minutes). (Note: A schedule window always needs to be shorter than the frequency of your search.)