www.fgks.org   »   [go: up one dir, main page]
Page MenuHomePhabricator
Create Task
Maniphest T345265

CommRel support for September 2023 Datacenter Switchover
Closed, ResolvedPublic
Actions

Description

Hi CommRel,

We are planning a datacenter switchover for the week of September 18th (week 38) with the following schedule:

The expected impact is 2-3 minutes of read-only on Wednesday, September 20th, 2023 14:00 UTC.

Note that we are implementing the changes described in Recurring, Equinox-based, Data Center Switchovers, in particular:

  • there is no switchback - we are staying in the Texas DC until the next switchover
  • future switchovers are predictable and happen every 6 months -- always on the week of an equinox

Let me know if you need more info on the changes.

Thank you!

Planning

As soon as the task is received by CRS

  • Ask SRE if anything major changed since the last time (noticeable things that are worth being announced).
    • If yes, update the process or the message.
  • Confirm when the wikis will be in read-only
  • Add the date to Asana, with a link to this task - done

Two weeks before (Sept 4 - week 36)

The week before (Sept 11 - week 37)

The week it happens (Sept 18 - week 38)

    • Emailing mailing lists:
      • wikitech-ambassadors once again (reminder)
      • wmfall (reminder, done by SRE)
  • Check with SRE if it goes well.

The week after (Sept 25 - week 39)

  • Debrief how it went, and document (comment on this task)

Details

Other Assignee
UOzurumba

Related Objects
Search...

Event Timeline

kamila created this task.Aug 30 2023, 3:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 30 2023, 3:05 PM
kamila moved this task from Incoming 🐫 to this.quarter 🍕 on the serviceops board.Aug 30 2023, 3:30 PM
kamila added a subscriber: MoveComms-Support (Jul-Sep-2023).Aug 30 2023, 4:30 PM
kamila added a project: MoveComms-Support (Jul-Sep-2023).Aug 31 2023, 12:41 PM
kamila updated the task description. (Show Details)
kamila updated the task description. (Show Details)
Trizek-WMF claimed this task.Aug 31 2023, 3:37 PM
Trizek-WMF moved this task from Backlog to Started on the MoveComms-Support (Jul-Sep-2023) board.
Trizek-WMF subscribed.

@kamila, thank you for asking for our support.

We have a message ready for communities. I invite you to review it as I will, and let me know if changes are needed: https://meta.wikimedia.org/wiki/Tech/Server_switch.

Then I will update the processes we have, to cover the new aspect of recurring switches.

Trizek-WMF triaged this task as High priority.Aug 31 2023, 3:37 PM
• RZamora-WMF subscribed.EditedAug 31 2023, 4:30 PM

@Trizek-WMF Thanks for claiming this Phab task 👍

kamila added a comment.Aug 31 2023, 4:43 PM

Thank you @Trizek-WMF ! The message looks good. Maybe I'd suggest replacing the word "first" with "primary" in the sentence "the switch between its first and secondary data centers", as from now on the datacenters are equal.

As an aside, can I help with translations? I am a native speaker of Slovak and that translation could use a bit of clean-up. I could also add a Czech translation.

Aklapper removed a subscriber: MoveComms-Support (Jul-Sep-2023).Sep 1 2023, 8:09 AM
• Elitre awarded a token.Sep 4 2023, 6:44 AM
Trizek-WMF added a comment.EditedSep 4 2023, 2:29 PM
In T345265#9134920, @kamila wrote:

Thank you @Trizek-WMF ! The message looks good. Maybe I'd suggest replacing the word "first" with "primary" in the sentence "the switch between its first and secondary data centers", as from now on the datacenters are equal.

Good catch! I edited the message accordingly, for a more neutral "The Wikimedia Foundation will switch the traffic between its data centers."

As an aside, can I help with translations? I am a native speaker of Slovak and that translation could use a bit of clean-up. I could also add a Czech translation.

Thank you for offering your help. Translations in Slovak are listed here. Please ping me on Slack if you need any assistance regarding translations.

kamila added a comment.Sep 4 2023, 2:32 PM

Thank you @Trizek-WMF, sounds good!

I will ping you regarding translations :-)

akosiaris awarded a token.Sep 4 2023, 2:40 PM
Trizek-WMF updated the task description. (Show Details)Sep 4 2023, 3:26 PM
Trizek-WMF updated the task description. (Show Details)Sep 4 2023, 3:44 PM
Trizek-WMF updated Other Assignee, added: UOzurumba.Sep 5 2023, 5:33 PM
Trizek-WMF updated the task description. (Show Details)Sep 6 2023, 2:03 PM
Trizek-WMF updated the task description. (Show Details)Sep 6 2023, 2:06 PM
Trizek-WMF updated the task description. (Show Details)
Trizek-WMF updated the task description. (Show Details)Sep 6 2023, 2:59 PM
Trizek-WMF updated the task description. (Show Details)Sep 11 2023, 3:32 PM
Trizek-WMF updated the task description. (Show Details)Sep 12 2023, 1:58 PM
Trizek-WMF updated the task description. (Show Details)Sep 12 2023, 2:04 PM
Trizek-WMF added a subscriber: UOzurumba.Sep 12 2023, 2:07 PM

I worked on some of the items for this week. I let @UOzurumba check on the done elements. Distribution to the village pumps will be done tomorrow when I won't be swamped with meetings.

@kamila, one action item is yours: tell the WMF about this upcoming event. Clément did it on Slack the last time. Don't forget to mention that SRE and CRS work together on this! ;)

UOzurumba updated the task description. (Show Details)Sep 12 2023, 2:14 PM
UOzurumba updated the task description. (Show Details)Sep 12 2023, 2:22 PM
kamila updated the task description. (Show Details)Sep 12 2023, 4:51 PM

Thank you for the reminder @Trizek-WMF! I sent a Slack announcement a while ago and now sent a reminder email as well.

I wrote "a number of different teams" for brevity, hope that's okay :D

Trizek-WMF updated the task description. (Show Details)Sep 15 2023, 9:38 AM
Trizek-WMF updated the task description. (Show Details)Sep 15 2023, 9:40 AM
UOzurumba updated the task description. (Show Details)Sep 18 2023, 3:59 PM
UOzurumba updated the task description. (Show Details)Sep 18 2023, 7:56 PM
UOzurumba updated the task description. (Show Details)Sep 18 2023, 9:30 PM
Trizek-WMF added a comment.Sep 20 2023, 2:15 PM

The banners were set and read: someone took the opportunity to post an article draft on the Global message talk page. :)

• RZamora-WMF awarded a token.Sep 20 2023, 6:39 PM
Trizek-WMF updated the task description. (Show Details)Sep 21 2023, 2:33 PM
Trizek-WMF lowered the priority of this task from High to Medium.Oct 2 2023, 1:01 PM

The post-event retro will be published later this week. Sorry for the delay.

@kamila, if you have any feedback regarding how we interacted and how the task went, please share it! :)

kamila added a comment.Oct 2 2023, 1:19 PM

Thanks a lot @Trizek-WMF !

I updated the parent task with a summary of how it went, feel free to use it in communications, and let me know if you need more detail :-)

Trizek-WMF added a comment.Oct 2 2023, 2:23 PM
In T345263#9215507, @kamila wrote:

The switchover went smoothly and had minimal user impact.

We haven't heard of disruptions flagged by community members.

Thanks as well to Community Relations who notified communities of the read-only window.

Thank you! :)

Trizek-WMF mentioned this in T345531: Switchover gitlab (gitlab1004 -> gitlab2002) - October 2023.Oct 3 2023, 1:09 PM
Trizek-WMF closed this task as Resolved.Oct 6 2023, 10:20 AM

Notes for CRS:

  • Update the template we have at Office wiki, as T345265 was improved "on the fly" t match the current switchover.
    • On planning task #1 "Ask SRE if anything major changed since the last time (noticeable things that are worth being announced)." it should be documented who to ask.
  • Document at Office Wiki all the conversations @UOzurumba and I had privately regarding translations.

Note for SRE:

Info for CRS and SRE:

  • We don't have any communication procedure in the case of our wikis being read-only longer than expected, or, worse, in a non-scheduled manner. I contacted the Social Media team to set up a process.
Trizek-WMF added a comment.Oct 6 2023, 10:32 AM

I'll take care of the CRS part, and follow up with SRE for the GitLab switch.

Trizek-WMF moved this task from Backlog to Done on the Datacenter-Switchover board.Oct 6 2023, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits