www.fgks.org   »   [go: up one dir, main page]

Page MenuHomePhabricator

Drop now unused user preferences from production database(s)
Open, Needs TriagePublic

Description

I'm really curious what happens to unused preferences left behind in the database by patches like https://gerrit.wikimedia.org/r/498903. There is no process to ever drop these, as far as I know. How can anyone identify these later? Do they clog up the database? Should we even care?

Possibilities:

  1. There is maintenance/cleanupPreferences.php. But do we ever run this? It looks like the script assumes all preferences have a default value specified somewhere. Is this really true? As far as I know it's possible to call setOption() without ever registering a default. Or what if an extension is temporarily disabled and we don't want the user's settings to immediately disappear?
  2. I also found maintenance/userOptions.php advancedsearch --delete which feels much saver to run on production, but leaves us with the job to identify the unused preferences first.
  3. I wonder if it's worth to actively drop unused preferences? Code could look like this:
// Purge obsolete BetaFeature option from the database
if ( $optionsManager->getOption( $user, 'advancedsearch' ) !== null ) {
	$optionsManager->setOption( $user, 'advancedsearch', null );
	$optionsManager->saveOptions( $user );
}

Possibly affected WMDE-TechWish projects:

Verified list

Note: Reference Previews and Two-Column-Edit-Conflict-Merge are still referencing their BetaFeatures flags in the code.

Event Timeline

Marostegui subscribed.

Removing the DBA tag as I am not sure if there's any actionable for us here. I believe this is about running a maintenance script to clean up things, right?
As long as it is done with the usual safe methods to avoid overloading the databases (in chunks, wait for replication etc), we (DBAs) are ok.

If I understood this the wrong way and this is an actionable for us, please let me know!

LSobanski added subscribers: Ladsgroup, LSobanski.

This looks like a Mediawiki question so while we (DBA) would likely be interested in the answer, I don't think we are the best team to provide one. I'll tag Platform Engineering and cc @Ladsgroup who may have an opinion.

(This comment was added in parallel to Manuel's so it's not a response to it)

Umherirrender subscribed.

The script is risky for wmf production and loops in dry-run mode (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/724784).

Some extensions depends on the behaviour of a non-defined default settings and not using the hook to exclude the deletion of the settings. I have learned that while working on T291748.

At the moment there is no clean up of the preferences, even the userjs part could be stored everything and forever. Users are also stored forever together with the non-defaults.

One option is to repeatly run a delete with LIMIT trough sql.php or such.
Or using userOptions.php to set the default value for the preference to remove. That script used the UserOptionsManager code which removes preferences equal to default. Maybe add a new option to delete just one option to make it faster to run.

MediaWiki also has no way for third party to remove all preferences if an extension gets uninstalled.

When I saw the ticket, my first impression was that "we haven't run this script in production for really long, it's probably broken in twenty different ways". It seems my hunch was correct as @Umherirrender gave a good explanation in details. I think it can be improved and then ran. That would help for sure.

I know @Pchelolo did some work in cleaning up user perfs storage and retrieval but I don't know how much they got done and how much is left.

I'd be happy to help this moved forward in code review and others but I admit this is not high priority from data persistence point of view. This table is rather small (fifteenth biggest table in enwiki, one tenth of the first one). So I don't think I will be able to spend time on it but would be more than happy to do consulting/review/deployment.

HTH

Change 929151 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/extensions/RevisionSlider@master] Use "0" instead of "false" for default user options

https://gerrit.wikimedia.org/r/929151

Change 929153 had a related patch set uploaded (by Thiemo Kreuz (WMDE); author: Thiemo Kreuz (WMDE)):

[mediawiki/extensions/AdvancedSearch@master] Use "0" instead of "false" for default user options

https://gerrit.wikimedia.org/r/929153

Change 929151 merged by jenkins-bot:

[mediawiki/extensions/RevisionSlider@master] Use "0" instead of "false" for default user options

https://gerrit.wikimedia.org/r/929151

Change 929153 merged by jenkins-bot:

[mediawiki/extensions/AdvancedSearch@master] Use "0" instead of "false" for default user options

https://gerrit.wikimedia.org/r/929153

Various subtasks of T54777: user_properties table bloat show some additional methods of cleaning up preferences: in addition to cleanupPreferences and userOptions --delete, we also have runBatchedQuery (T66754), a custom temporary single-purpose maintenance script (T304461), or manual DELETE statements on the primary (T308309).

Mentioned in SAL (#wikimedia-operations) [2024-05-11T12:07:21Z] <Dreamy_Jazz> Running foreachwiki userOptions.php --delete templatewizard-betafeature - T300371

Mentioned in SAL (#wikimedia-operations) [2024-05-11T13:11:12Z] <Dreamy_Jazz> Running foreachwiki userOptions.php --delete betafeatures-popup-disable - T300371

Mentioned in SAL (#wikimedia-operations) [2024-05-11T13:14:42Z] <Dreamy_Jazz> Running foreachwiki userOptions.php --delete betafeatures-vector-typography-update --nowarn - T300371

Mentioned in SAL (#wikimedia-operations) [2024-05-11T13:17:24Z] <Dreamy_Jazz> Running foreachwiki userOptions.php --delete betafeatures-vector-compact-personal-bar --nowarn - T300371

Mentioned in SAL (#wikimedia-operations) [2024-05-11T13:20:28Z] <Dreamy_Jazz> Running foreachwiki userOptions.php --delete betafeatures-geonotahack --nowarn - T300371

User properties over 1M rows in enwiki, maybe that'd be helpful for clean up:

mysql:research@s1-analytics-replica.eqiad.wmnet [enwiki]> select up_property, count(*) from user_properties group by up_property order by count(*) desc limit 100;
+-----------------------------------------------------+----------+
| up_property                                         | count(*) |
+-----------------------------------------------------+----------+
| echo-subscriptions-email-page-review                | 19153xxx |
| echo-subscriptions-email-edit-thank                 | 19033xxx |
| echo-subscriptions-web-article-linked               | 18891xxx |
| echo-subscriptions-email-mention                    | 18869xxx |
| echo-subscriptions-web-reverted                     | 18845xxx |
| echo-subscriptions-email-article-linked             | 18493xxx |
| popups                                              | 13718xxx |
| skin                                                | 10012xxx |
| thumbsize                                           |  9307xxx |
| rememberpassword                                    |  8187xxx |
| visualeditor-autodisable                            |  8005xxx |
| visualeditor-hidebetawelcome                        |  6998xxx |
| VectorSkinVersion                                   |  4377xxx |
| echo-subscriptions-email-dt-subscription            |  3708xxx |
| watchcreations                                      |  3468xxx |
| growthexperiments-homepage-pt-link                  |  2730xxx |
| growthexperiments-homepage-enable                   |  2730xxx |
| growthexperiments-help-panel-tog-help-panel         |  2717xxx |
| growthexperiments-homepage-variant                  |  2714xxx |
| growthexperiments-tour-help-panel                   |  2714xxx |
| growthexperiments-tour-homepage-mentorship          |  2704xxx |
| discussiontools-autotopicsub                        |  2575xxx |
| watchlisttoken                                      |  2527xxx |
| echo-subscriptions-web-wikibase-action              |  2492xxx |
| timecorrection                                      |  2320xxx |
| date                                                |  2217xxx |
| growthexperiments-homepage-mentorship-enabled       |  2099xxx |
| nickname                                            |  2048xxx |
| visualeditor-editor                                 |  1910xxx |
| growthexperiments-tour-homepage-discovery           |  1701xxx |
| homepage_mobile_discovery_notice_seen               |  1619xxx |
| gettingstarted-task-toolbar-show-intro              |  1351xxx |
| growthexperiments-tour-homepage-welcome             |  1294xxx |
| visualeditor-hideusered                             |  1260xxx |
| enotifusertalkpages                                 |  1127xxx |
| growthexperiments-homepage-se-filters               |  1089xxx |
| growthexperiments-tour-newimpact-discovery          |  1027xxx |
| variant                                             |  1015xxx |
| compact-language-links                              |  1008xxx |