Skip to content

Storage providers, server and global variables, multi-server sync and dialog canvases - #275

Open
1robie wants to merge 23 commits into
Maxlego08:developfrom
1robie:feature/storage-restructure
Open

1robie wants to merge 23 commits into
Maxlego08:developfrom
1robie:feature/storage-restructure

Conversation

@1robie

@1robie 1robie commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replaces zMenu's single SQL storage with pluggable storage providers, moves player data into a variables table shared with new server and global variables, and lets several servers share one database safely. Also adds dialog canvases and rewrites the packetevents hook. Existing data is copied over on the first start and the old tables are kept; the old player data API keeps working and is deprecated (since = "1.1.2.2", forRemoval = true) in favour of VariableService.

Storage providers

  • storage-type now selects a StorageProvider (experimental SPI in fr.maxlego08.menu.api.storage.spi), registered in the new STORAGE_PROVIDER registry. Addons can add their own backend.
  • Built-in: SQLITE, MYSQL, MARIADB, POSTGRESQL, REDIS (standalone or Cluster), MONGODB, and NONE (in memory until the server stops). The Redis, PostgreSQL and MongoDB drivers are downloaded only when config.yml asks for them.
  • Writes run one at a time, in the order they were made; reads run on their own threads.
  • /zm storage migrate <type> copies every zMenu data to another storage (confirm by running it twice). migration-database-configuration gives the target its own connection, for example MariaDB to PostgreSQL. /zm storage status shows the storage in use.
  • database-configuration.properties passes driver properties (sslmode, allowPublicKeyRetrieval, ...).

Variables

  • The data action takes scope: player (default), server (one value per server-id) or global (shared by every server). Placeholders: %zmenu_server_value_<key>%, %zmenu_global_value_<key>% and the _key_exist_, _expire_, _expire_format_, _is_expired_ variants. Commands: /zm variables <get|set|add|subtract|remove|keys> <server|global> ....
  • add and subtract send the amount to the database, which adds it (SQL COALESCE + ?, Redis HINCRBYFLOAT, MongoDB $inc), so two servers adding at once both count. Decimals are kept. An addition to an expired value starts again from 0.
  • Player data is read just before a player joins and kept while they are online. Offline lookups are read in the background and kept for 5 minutes. Nothing is read for every player at startup any more. Changes are batched every batch-task seconds and merged per variable.
  • Expired values are purged every 10 minutes.

Multi-server

  • Servers sharing a database tell each other about changes: an events table polled every messaging-poll-interval seconds (messaging-type: AUTO/SQL), Redis pub/sub (REDIS), or nothing (NONE). A server re-reading a changed value never hides a write of its own that is not stored yet.
  • Player sessions: a player joining a server waits up to session-handoff-timeout ms for the server they left to store their data. A refused login releases the session. Servers heartbeat, and a duplicate server-id is reported.
  • Saved inventories (clear-inventory) are kept per server in saved_inventories, so items taken on one server are only given back there.

Data migration on first start

  • player_datas is copied into variables and player_inventories into saved_inventories (to the first server that starts), each in one transaction; the old tables are untouched.
  • Requires Sarah 1.27.

Dialog canvases

  • dialog-canvas body: images and text at pixel positions inside a dialog (Paper 1.21.6+). Images come from resource-pack/textures, vanilla, Nexo, ItemsAdder or CraftEngine; text uses the vanilla font or resource-pack/fonts.yml.
  • Canvas API (fr.maxlego08.menu.api.dialog.canvas, MenuPlugin#getCanvasManager()) and the GLYPH_PROVIDER, CANVAS_PACK_TARGET, CANVAS_ELEMENT registries.
  • Dialog custom-click buttons use a per-player click table instead of Paper click callbacks kept for 12 hours.

packetevents hook

  • Rewritten around one packet listener and one session per player, shared by the title, title animation, hidden inventory and click limiter features.
  • Fixes clear-inventory-type: PACKET_EVENT showing the real inventory after a shift-click or resync, show-items-after-animation showing items too early, items looking lost after a blocked fast click, and listeners left registered after a reload.

Fixes

  • A saved inventory given back on shutdown is removed from the database (no second copy on the next join).
  • Visited worlds and played animations are read back correctly. They always came back empty, so first-join items and one-time animations repeated.
  • Visited worlds are no longer read on the main thread at every join.
  • MySQL/MariaDB deadlocks between two servers creating the same value are retried.

Compatibility

  • DataManager, PlayerData, Data, the player data methods of StorageManager and the storage.dto records are deprecated and still work. DataManager#getPlayer only returns players in memory; use loadPlayer(UUID) for offline players.
  • The ItemsAdder hook now targets ItemsAdder 4; ItemsAdder 3 is no longer supported.

Tests

  • Contract suites for every provider, two-server suites (handoff, shared increments, clears), and migration tests. They pass on SQLite, memory and embedded Redis/MongoDB, and against real MySQL 8.4, MariaDB 11, PostgreSQL 16, Redis 7 and MongoDB 7 (enabled with -Dzmenu.it.<backend>=host:port, see RealDatabases).
  • run-paper task for a local test server.

1robie added 23 commits October 27, 2026 10:01
…anvas

# Conflicts:
#	Common/src/main/java/fr/maxlego08/menu/inventory/inventories/InventoryDefault.java
#	changelog.md
…d visited worlds async

- Run every database write on a single writer thread so inserts, upserts and deletes
  apply in the order they were made; flush and close it on disable
- Delete the saved inventory of players restored on shutdown so it is not given again
  on their next join
- Read first-world-join visited worlds off the main thread once per session
- Use concurrent maps for player data and saved inventories
- Add the experimental storage SPI (StorageProvider, stores, capabilities) and the
  STORAGE_PROVIDER registry so addons can add backends
- Move the SQL code into SqlStorageProvider (SQLite, MySQL, MariaDB) on the existing
  tables, with an ordered writer thread and a reader pool; close the pool on disable
- Add MemoryStorageProvider for storage-type NONE and MockBukkit servers
- ZStorageManager now wraps the chosen provider; StorageManager behaves as before
- Replace GlobalDatabaseConfiguration with StorageConfiguration and consistent defaults
- Read visited worlds and played animations by column name; both always came back empty
- Saving a player inventory replaces the previous copy
- Add provider contract tests for SQLite and memory
- Read a player's data and saved inventory before they join (AsyncPlayerPreLoginEvent)
  instead of every row at startup; refuse the login when they cannot be read
- Keep offline players looked up by commands or placeholders for 5 minutes, then evict
  them once nothing of theirs waits to be written
- Add DataManager#loadPlayer; /zm players commands load offline targets first
- Replace the player data write buffer with a map keyed by player and key, and lay
  pending values over loads
- List data keys with a DISTINCT query; purge expired data every 10 minutes
- Storage SPI: PlayerDataStore#keys and #deleteExpired, SavedInventoryStore#load
- Add MockBukkit and provider tests
- Add the variable API (VariableScope, VariableRef, Variable, VariableService) and
  MenuPlugin#getVariableService
- Store every scope in a new variables table with typed number values; copy the old
  player_datas rows once, in one transaction
- VariableStore replaces PlayerDataStore in the storage SPI; increments run in the
  database (UPDATE ... COALESCE + ?, then an atomic upsert)
- ZVariableService caches owners: server/global at startup, players lazily; player
  changes are batched and merged (increments add up), server/global written at once
- DataManager and PlayerData now sit on the player scope of the service
- data action and button datas take scope: player|server|global; add/subtract,
  /zm players add|subtract and the MythicMobs mechanic use increments
- Add %zmenu_server_*% and %zmenu_global_*% placeholders, /zm variables and server-id
- Switch to Sarah 1.27-local (mavenLocal limited to fr.maxlego08.sarah)
- Add MessageBroker, SessionStore and StorageEvent to the storage SPI, plus
  StorageContext#serverId and VariableStore#load
- SQL provider: storage_events table polled by every server (overlap window for ids
  committed out of order), player_sessions and servers tables
- ZVariableService publishes player/global changes and clears, re-reads what other
  servers change (keeping its own pending changes on top), and hands players over:
  the joining server waits for the old one to store and release the player
- Heartbeat every 10 s; report another running server with the same server-id
- Saved inventories are kept per server (saved_inventories), legacy rows copied once
- New config: messaging-type, messaging-poll-interval, session-handoff-timeout
- Add two-server tests on a shared SQLite database
- RedisStorageProvider (storage-type: REDIS) on Jedis, standalone or Cluster: variables
  in per-owner hashes sharing a hash tag, Lua scripts for set/delete/increment
  (HINCRBYFLOAT), name counts, expiry sorted set, saved inventories per server, world
  visits, animations, a capped open-inventory stream and sessions
- RedisMessageBroker on pub/sub, also usable with a SQL store (messaging-type: REDIS)
- Download Jedis at runtime only when storage-type or messaging-type is REDIS; the
  provider class names no Jedis type so zMenu loads without it
- Warn (or refuse with redis.require-aof) when Redis has appendonly off
- Share the ordered writer and reader threads between providers (StorageThreads)
- Run the storage contract and two-server tests on an in-process Redis (jedis-mock)
- StorageMigration copies variables of every scope, saved inventories, visited worlds
  and played animations from one provider to another, in pages
- /zm storage migrate <type> (confirmed by a second run) opens a fresh provider for the
  target with the config.yml settings; /zm storage status shows the storage in use
- Storage SPI: EXPORT capability, VariableStore#owners, WorldVisitStore#players and
  AnimationStore#players (default to a failed future); implemented in SQL, Redis, memory
- storage-type POSTGRESQL through Sarah's PostgreSQL dialect; the loader downloads the
  PostgreSQL driver and Jedis for storage-type, messaging-type or migration-target
- Test a SQLite to Redis to memory copy that keeps every value
- MongoStorageProvider (storage-type: MONGODB) on the sync driver: one document per
  variable with Decimal128 numbers and atomic $inc, saved inventories per server, world
  visits, animations, the open-inventory log and sessions
- MongoPollingBroker: events collection numbered by a counter document, read with the
  same overlap window as the SQL broker
- Download the MongoDB driver only when storage-type or migration-target is MONGODB; the
  provider class names no driver type
- Share the event <-> fields conversion between the SQL, Redis and MongoDB brokers
- Run the storage contract, two-server and migration tests on mongo-java-server
- Deprecate for removal (since 1.1.2.2) DataManager, PlayerData, Data,
  MenuPlugin#getDataManager, the player data methods of StorageManager and
  loadInventories, the storage.dto records and Tables.PLAYER_DATAS/PLAYER_INVENTORIES
- Javadoc points each one to its VariableService or storage SPI replacement
… Redis and MongoDB

- Opt-in real-server tests (-Dzmenu.it.<backend>=host:port): the storage contract, the
  two-server contract and a copy through every backend; the build forwards the properties
- Merge the SQLite, Redis and MongoDB two-server tests into TwoServerContractTest
- Retry a SQL variable transaction cancelled by a deadlock or serialization conflict
  (40001/40P01): two servers creating the same row deadlocked on MySQL and MariaDB
- StorageMigration returns once every write is stored in the target
- Polling brokers wait for a read in progress when they stop
When the limiter cancelled a CLICK_WINDOW packet, the client kept its
predicted result and the clicked item looked gone until the menu
reopened. The player's inventory is now resent after a blocked click.

The limiter also kept players in its set after switching to a menu with
click-limiter-enabled: false, after dying with a menu open, or after
disconnecting on Folia. It now clears them in each of these cases.
…connection

database-configuration.properties passes driver properties to MySQL, MariaDB and PostgreSQL. migration-database-configuration gives /zm storage migrate its own connection settings, so data can be copied between two SQL databases. A failed database connection no longer throws a second error while enabling, and a failed migrate logs the database error once.
The hook now has one packet listener (PacketRouter), one inventory
listener (PacketInventoryBridge) and one PacketSession per player, shared
by the title, title animation, hidden inventory and click limiter
features. The old listeners, FakeInventory, PacketListener and
fakeContents are removed. The public API and the menu options are
unchanged.

Fixes:
- clear-inventory-type: PACKET_EVENT no longer shows the real inventory
  after a shift-click, a blocked click or a resync: every packet that
  carries a player slot is rewritten while the menu is open
- show-items-after-animation now holds the items until the title
  animation ends; the window content is held on the network thread
  instead of cancelled too late
- An addition to an expired value that is not purged yet now starts
  from 0 with the new expiry time, in every store (SQL, Redis, MongoDB,
  memory). It used to add to the stale value and keep the old expiry,
  so the next purge deleted it.
- set and delete on an offline player now read their variables, so the
  next batch writes the change instead of holding it until shutdown.
- increment on an offline player completes with the stored value plus
  the amount, not the amount alone.
- A value read for another server's change no longer hides a write of
  this server that was not stored yet: the read waits for it, or is
  made again when such a write ran meanwhile.
- The session claimed at pre-login is released when the login is
  refused, or when the player has not joined after 60 seconds; joining
  claims it again. A saved inventory read for a refused login is
  forgotten.
- /zm storage migrate opens the target with config.yml, then
  database-configuration.yml, then migration-database-configuration, so
  the shared file no longer replaces the migration values. Migrating to
  the storage in use is refused by backend, address, database and table
  prefix rather than by the storage-type text.
- Redis deleteName sends 500 owners per round trip.
- VariableService gains ownerOf and ref, used by the data action.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant