Pewarta Proxy

📋 Server Request & Proxy Logs

Loading active proxy settings...
Developed by Khairul Adha (adhexa.id)
Mengambil log server...

⚙️ Pengaturan Proxy Target

Skip to content

fix(mysql): read and compare text as utf8mb4; pre-scan sees collation, comment, generated - #268

Merged
jasdeepkhalsa merged 1 commit into
masterfrom
fix/mysql-utf8mb4
Oct 12, 2026
Merged

jasdeepkhalsa merged 1 commit into
masterfrom
fix/mysql-utf8mb4

Conversation

@jasdeepkhalsa

Copy link
Copy Markdown
Member

Found by testing multilingual data (Gurmukhi, transliterations, translations in Hindi, Urdu and Chinese, emoji, Brahmi) under mixed column, table and database collations on MySQL 8.0 and Dolt 2.2.3.

1. Four-byte characters: changes were silently missed (both data paths)

  • DBDiff connected to MySQL as utf8, which in MySQL is utf8mb3 (at most three bytes a character).
  • It also compared rows through CAST(… AS CHAR CHARACTER SET utf8) / CONVERT(… USING utf8).
  • Anything needing four bytes (emoji, and scripts outside the Basic Multilingual Plane) became ?. So 🙏 → 🙂 or 𑀅𑀓 → 𑀅𑀔 produced no migration at all.
  • Gurmukhi and other BMP scripts were unaffected.
  • Fix: the connection is utf8mb4 / utf8mb4_unicode_ci, and every comparison casts to utf8mb4. (The migration runner's MySQL connection already used utf8mb4.)

2. The schema pre-scan skipped real changes

  • The pre-scan fingerprint hashed each column's type, default, nullability and extras.
  • It left out character set, collation, comment and generated expression.
  • So a table differing only in one of those was reported "identical", e.g. a column moved from utf8mb4_unicode_ci to utf8mb4_0900_ai_ci, a changed column comment, or a changed GENERATED ALWAYS AS.
  • Fix: they're now in the fingerprint.

Verified to work, byte for byte, same-server and cross-server

  • Gurmukhi text changes, and Gurmukhi primary keys.
  • Case-only changes in case-insensitive columns.
  • NFC vs decomposed accents (look identical, different bytes).
  • latin1 columns, and table-level collation changes.
  • Numeric-looking text keys.

New test: tests/DataDiff/UnicodeMySQLTest.php (MySQL and Dolt jobs). All three cases fail on master and pass here.

  • Same-server via the CLI: the multilingual fixture's generated UP is applied, and the target must equal the source byte for byte (HEX of every value), collations included.
  • Streaming merge (cross-server path): the same rows, including a row whose only difference is a four-byte character.
  • Pre-scan: collation-only, comment-only (in Gurmukhi) and generated-expression-only changes are each detected.

Dolt quirks found (not DBDiff bugs); the test accounts for both:

  • Dolt silently skips an UPDATE that changes only case in a case-insensitive column ("0 rows affected"), so on Dolt the test checks the planned SQL rather than the applied bytes.
  • Dolt leaves information_schema.COLUMNS.GENERATION_EXPRESSION empty, so the pre-scan can't see a generated-expression-only change there.

Not fixable in DBDiff: MySQL stores column comments in utf8mb3, so an emoji in a comment is already ? in MySQL itself.

Locally, all passing:

Suite Result
MySQL 8.0 DBDiff,Unit passed
Dolt DBDiff,Unit passed
Postgres suite passed
SQLite passed

🤖 Generated with Claude Code

…, comment, generated

DBDiff connected to MySQL as utf8 (utf8mb3) and compared rows cast to it,
so a four-byte character — an emoji, a script outside the Basic
Multilingual Plane — read as '?', and a change between two of them was
not seen. The schema pre-scan left out a column's character set,
collation, comment and generated expression, so a table differing only
there was skipped as unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added bug mysql Related to Mysql php Pull requests that update php code postgres Related to Postgres sqlite Related to Sqlite labels Oct 12, 2026
@sonarqubecloud

Copy link
Copy Markdown

@jasdeepkhalsa
jasdeepkhalsa merged commit 3e3f9dc into master Oct 12, 2026
81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug mysql Related to Mysql php Pull requests that update php code postgres Related to Postgres sqlite Related to Sqlite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant