|
| 1 | +# Adding a new database dialect |
| 2 | + |
| 3 | +A walkthrough of every file you need to touch, in roughly the order you should touch them. The |
| 4 | +SQLite support PR is the cleanest recent reference — every step below points at a SQLite analogue. |
| 5 | + |
| 6 | +## 0. What the upstream `foundations-jdbc` library must already provide |
| 7 | + |
| 8 | +`typr` is the code generator; the runtime types and codecs live in the external |
| 9 | +`dev.typr.foundations:foundations-jdbc` library. Before you start, make sure that library exposes: |
| 10 | + |
| 11 | +- A typed types catalogue, e.g. `dev.typr.foundations.XxxTypes` with one entry per declarable type |
| 12 | + (integer/varchar/decimal/date/…). Each entry is a `XxxType<JvmType>` carrying the read/write |
| 13 | + codec and JSON codec. |
| 14 | +- `dev.typr.foundations.connect.XxxConfig` for building a JDBC connection, and |
| 15 | + `DatabaseKind.XXX` enum value. |
| 16 | +- Scala and Kotlin wrappers: `dev.typr.foundationssc.XxxTypes` and |
| 17 | + `dev.typr.foundationskt.XxxTypes`. |
| 18 | + |
| 19 | +If any of these are missing, send the PR to foundations first. |
| 20 | + |
| 21 | +## 1. Add a `Dialect` entry |
| 22 | + |
| 23 | +File: `typr-dsl/src/java/dev/typr/dsl/Dialect.java` — the DSL's per-dialect SQL grammar bits |
| 24 | +(identifier quoting, casts, null-safe comparison, LIMIT/OFFSET, tuple-IN support). |
| 25 | + |
| 26 | +The `Dialect` interface has reasonable defaults for everything except `bigint()`, `quoteIdent()`, |
| 27 | +`escapeIdent()`, `typeCast()`, `columnRef()`, `nullSafeEquals()`, `nullSafeNotEquals()` — override |
| 28 | +just what differs from the default (PostgreSQL-shaped) behaviour. |
| 29 | + |
| 30 | +Re-export it in the Scala wrapper at |
| 31 | +`typr-dsl-scala/src/scala/dev/typr/dslsc/package.scala`'s `object Dialect`. |
| 32 | + |
| 33 | +(Kotlin can use the Java static field directly — no wrapper needed.) |
| 34 | + |
| 35 | +## 2. Define the `db.XxxType` ADT |
| 36 | + |
| 37 | +File: `typr-codegen/src/scala/typr/db.scala`. |
| 38 | + |
| 39 | +Add a `sealed trait XxxType extends Type` plus one `case object` (or `case class` for parameterised |
| 40 | +types like `Decimal(precision, scale)`) per concrete declarable type. Add `XxxType` to the |
| 41 | +`Unknown` mixin trait list at the bottom — that's the fallback for unrecognised JDBC type names. |
| 42 | + |
| 43 | +## 3. Add `DbType.Xxx` + connection plumbing |
| 44 | + |
| 45 | +Files: |
| 46 | + |
| 47 | +- `typr-codegen/src/scala/typr/DbType.scala` — add `case object Xxx extends DbType` returning your |
| 48 | + adapter, plus a branch in `detect(...)` and `detectFromDriver(...)`. |
| 49 | +- `typr-codegen/src/scala/typr/TypoDataSource.scala` — add a `hikariXxx*(...)` constructor and a |
| 50 | + `DatabaseKind.XXX` branch in `hikari(...)`. |
| 51 | + |
| 52 | +## 4. Write the codegen adapter |
| 53 | + |
| 54 | +File: `typr-codegen/src/scala/typr/internal/codegen/XxxAdapter.scala`. |
| 55 | + |
| 56 | +Mirror `DuckDbAdapter` (closest in spirit for most embedded DBs) or `Db2Adapter` (single-schema, |
| 57 | +no native arrays). Five layers: |
| 58 | + |
| 59 | +1. **SQL syntax** — `quoteIdent`, `typeCast`, the various `columnReadCast`/`columnWriteCast` (most |
| 60 | + dialects can return `Code.Empty`). |
| 61 | +2. **Runtime types** — point at `XxxTypes` / `XxxType` / `XxxText` and pick a `typeFieldName` |
| 62 | + (e.g. `xxxType`). |
| 63 | +3. **Capabilities** — `supportsArrays`, `supportsReturning`, `supportsCopyStreaming`, the upsert |
| 64 | + strategy. |
| 65 | +4. **SQL templates** — upsert (`ON CONFLICT` vs `MERGE`), conflict update clause, returning clause. |
| 66 | +5. **Schema DDL** — `dropSchemaDdl` / `createSchemaDdl`. For dialects without schemas (SQLite, |
| 67 | + embedded DBs), return a SQL comment. |
| 68 | + |
| 69 | +## 5. Write the metadata-extraction package |
| 70 | + |
| 71 | +Directory: `typr-codegen/src/scala/typr/internal/xxx/`. Four files: |
| 72 | + |
| 73 | +- `XxxJdbcMetadata.scala` — wraps `ResultSetMetaData` into `MetadataColumn`. This is essentially |
| 74 | + identical across dialects — copy `DuckDbJdbcMetadata` and rename. |
| 75 | +- `XxxTypeMapperDb.scala` — maps the declared/JDBC type name string to a `db.XxxType`. Read your |
| 76 | + database's reference for type aliases; many dialects accept synonyms (BIGINT/INT8/LONG/…). |
| 77 | +- `XxxSqlFileMetadata.scala` — analyses user-supplied `.sql` files using sqlglot. Copy |
| 78 | + `DuckDbSqlFileMetadata`; the only dialect-specific bits are (a) how you read the schema (the |
| 79 | + catalogue queries) and (b) the sqlglot `dialect` string. |
| 80 | +- `XxxMetaDb.scala` — reads tables/columns/PKs/FKs/uniques/views from the database catalogue. |
| 81 | + Look at how your DB exposes metadata (information_schema, system catalog views, or PRAGMAs) and |
| 82 | + shape the queries to fit. Return a `MetaDb(dbType, relations, enums = Nil, domains = Nil, …)`. |
| 83 | + |
| 84 | +## 6. Wire all dispatch sites |
| 85 | + |
| 86 | +These are the files that have one `case DbType.X => …` arm per dialect. Each new dialect needs an |
| 87 | +arm added everywhere: |
| 88 | + |
| 89 | +- `typr-codegen/src/scala/typr/MetaDb.scala` — both the `typeMapperDb` match and the `fromDb` |
| 90 | + match. |
| 91 | +- `typr-codegen/src/scala/typr/internal/InstanceRequirements.scala` — the heuristic name guess |
| 92 | + and the `dbTypeFieldNameFor` map. |
| 93 | +- `typr-codegen/src/scala/typr/internal/generate.scala` — the `databaseName` string used as a |
| 94 | + TypeDefinition discriminator, **plus** the precision-types case list near the bottom that emits |
| 95 | + `PreciseConstraint.*` for `VarChar(Some(n))` etc. |
| 96 | +- `typr-codegen/src/scala/typr/internal/codegen/DbLibFoundations.scala` — selectByIds / |
| 97 | + deleteByIds bodies. If your DB has no native arrays (most non-Postgres do), fold it into the |
| 98 | + existing `case DbType.SqlServer | DbType.DB2 | …` arms rather than copy-pasting. |
| 99 | +- `typr-codegen/src/scala/typr/internal/sqlfiles/SqlFileReader.scala` — dispatch to your |
| 100 | + `XxxSqlFileMetadata`. |
| 101 | +- `typr-codegen/src/scala/typr/internal/TypeMapperJvmNew.scala` — both the `baseType` and the |
| 102 | + precise-types match. (`TypeMapperJvmOld` is PostgreSQL-only legacy; skip.) |
| 103 | +- `typr-codegen/src/scala/typr/internal/TypeMatcher.scala` — `typeName` (for the matcher's |
| 104 | + per-column name string). |
| 105 | +- `typr-codegen/src/scala/typr/internal/TypeCompatibilityChecker.scala` — add a `CompatibilityClass` |
| 106 | + arm for every Java-equivalent class (String/Boolean/Int/Long/…) so cross-dialect Bridge type |
| 107 | + matching works. |
| 108 | +- `typr-codegen/src/scala/typr/internal/ComputedTestInserts.scala` — the `case`s for max-length |
| 109 | + detection on text columns. |
| 110 | +- `typr/src/scala/typr/bridge/TypeSuggester.scala` — group types into the "text/integer/numeric/ |
| 111 | + boolean/temporal/uuid/json" buckets the bridge UI uses. |
| 112 | + |
| 113 | +## 7. CLI wiring |
| 114 | + |
| 115 | +- `typr-config.schema.json` — add `"xxx"` to the boundary `type` enum and add a `xxxBoundary` |
| 116 | + definition (use `duckdbBoundary` as a template for embedded DBs, or `databaseBoundary` for |
| 117 | + server-based ones). |
| 118 | +- Run `bleep run generate-config-types` to regenerate the typed config classes |
| 119 | + (`XxxBoundary.scala` shows up under `typr-codegen/generated-and-checked-in-jsonschema/`). |
| 120 | +- `typr/src/scala/typr/cli/config/ConfigParser.scala` — add `Some("xxx") => …` arms for source |
| 121 | + *and* boundary parsing, and add `ParsedSource.Xxx` / `ParsedBoundary.Xxx` cases. |
| 122 | +- `typr/src/scala/typr/cli/config/ConfigToOptions.scala` — `convertXxxBoundary` + `convertXxxSource`. |
| 123 | +- `typr/src/scala/typr/cli/commands/Generate.scala` — `fetchXxxBoundary`, `fetchXxxSource`, |
| 124 | + `generateXxxForOutput`, plus the two dispatch sites in `runTwoPhaseGeneration`. Watch for any |
| 125 | + driver-specific quirks loading the schema (SQLite's xerial driver, for example, can't run |
| 126 | + multi-statement strings in a single `execute()` — the SQLite path splits on `;` first). |
| 127 | +- `typr/src/scala/typr/cli/app/MetaDbFetch.scala` — add a `ParsedSource.Xxx` arm + `fetchXxx`. |
| 128 | +- `typr/src/scala/typr/cli/app/ConnectionTest.scala` — add a `ParsedSource.Xxx` arm + `tryXxx`, |
| 129 | + and include `Xxx` in `isTestable`'s pattern. |
| 130 | +- `typr/src/scala/typr/cli/app/LoadedSource.scala` — include `ParsedSource.Xxx` in the database |
| 131 | + pattern arm. |
| 132 | +- The TUI screens (`SchemaPicker`, `SourceForm`, `SourceList`, `MainMenu`) — fold the new kind |
| 133 | + string into the existing `"duckdb" | "xxx"` cases for path-style sources, or use the database |
| 134 | + patterns for host/port sources. |
| 135 | + |
| 136 | +## 8. Build wiring |
| 137 | + |
| 138 | +- `bleep.yaml` — add three tester project entries (`testers/xxx/java`, `testers/xxx/kotlin`, |
| 139 | + `testers/xxx/scala`), each with the right JDBC driver dependency. Also add the driver to the |
| 140 | + `typr-codegen` project's dependencies so the CLI can connect. |
| 141 | +- `typr.yaml` — add a boundary entry (with `schema_sql`, `sql_scripts`, `path` or host/port) and |
| 142 | + three outputs (`xxx-java`, `xxx-kotlin`, `xxx-scala`). |
| 143 | + |
| 144 | +## 9. Test data + scripts |
| 145 | + |
| 146 | +- `sql-init/xxx/00-schema.sql` — exercise every column type your `db.XxxType` ADT models, plus |
| 147 | + composite PK, composite FK, UNIQUE, views, and `precision_types[_null]` for precise-type |
| 148 | + generation. |
| 149 | +- `sql-scripts/xxx/*.sql` — half a dozen parameterised queries covering SELECT, INSERT-with- |
| 150 | + RETURNING, UPDATE, DELETE, JOIN. |
| 151 | + |
| 152 | +## 10. Testers |
| 153 | + |
| 154 | +Under `testers/xxx/{java,kotlin,scala}` add a `src/{lang}/testdb/XxxTestHelper` and a |
| 155 | +`BasicCrudTest` that exercises the generated repos. Mirror `testers/duckdb/{java,kotlin,scala}` — |
| 156 | +if your driver supports multi-statement `execute()`, you can use `connectionInitSql(schema)` |
| 157 | +directly; otherwise split the schema yourself (SQLite shows this pattern). |
| 158 | + |
| 159 | +## 11. Generate, fmt, test |
| 160 | + |
| 161 | +```bash |
| 162 | +bleep run typr -- generate --source xxx --accept |
| 163 | +bleep fmt |
| 164 | +bleep test testers/xxx |
| 165 | +``` |
| 166 | + |
| 167 | +## Things that bit me on SQLite specifically |
| 168 | + |
| 169 | +- **Single connection, in-memory.** `:memory:` databases live inside one JDBC connection; opening |
| 170 | + a second connection gives you an empty DB. Foundations' `singleConnectionMode()` handles the |
| 171 | + reuse but `connectionInitSql` runs once via a *single* `stmt.execute(...)` — for drivers that |
| 172 | + don't accept multi-statement strings (xerial SQLite is one) the schema only partially loads. |
| 173 | + The fix is to bypass `connectionInitSql` and run statements one at a time during a one-shot |
| 174 | + init, then switch the transactor to `rollbackOnly()` for tests. |
| 175 | +- **No schemas.** Force `SchemaMode.SingleSchema("main")` in your `convertXxxBoundary`, and emit |
| 176 | + `RelationName(None, name)` in metadata. SQLite's `main` namespace isn't really a schema. |
| 177 | +- **Type affinity, not declared type.** SQLite stores values in five storage classes regardless |
| 178 | + of column declaration — your `XxxTypeMapperDb` needs to match common synonyms (BIGINT, INT8, |
| 179 | + INT2, VARCHAR(n), CLOB, …) to a specific `db.XxxType` so the codecs round-trip. The full |
| 180 | + affinity-substring fallback at the bottom of `SqliteTypeMapperDb` handles "anything goes" cases |
| 181 | + (`VARYING CHARACTER`, `DOUBLE PRECISION`, etc.). |
| 182 | +- **Foreign keys off by default.** SQLite needs `PRAGMA foreign_keys = ON` per connection. The |
| 183 | + foundations `SqliteConfig.Builder.foreignKeys(true)` sets it via a driver property, but if |
| 184 | + you're opening raw JDBC for tests, set it yourself. |
0 commit comments