Add MySQL DBI config translator and pipeline orchestrator - #2196
Open
Klarsen04 wants to merge 25 commits into
Open
Add MySQL DBI config translator and pipeline orchestrator#2196Klarsen04 wants to merge 25 commits into
Klarsen04 wants to merge 25 commits into
Conversation
Create the core MySQL receiver translator with functional options pattern: - translator.go with endpoint, username, passfile, TLS, localhost detection - translator_test.go with unit tests for all option functions - Register mysqlreceiver.NewFactory() in defaultcomponents - Add MySQLKey and MySQL DBI component name constants to common.go - Add go.mod dependency on mysqlreceiver
Add YAML configs and translator wiring for MySQL DBI pipelines: - dbi_dbload_mysql.yaml: count connector for active sessions by wait/user/db/sql/host - dbi_topsql_mysql.yaml: signaltometrics connector for top query metrics - transform_dbi_fix_start_time_mysql.yaml: transform processor for MySQL timestamps - Update count, signaltometrics, and transform translators to handle MySQL component names - Add unit tests for MySQL connector/processor creation
Add mysql[] array schema to database_insights in schema.json: - endpoint, instance_name, username, password_file - tls.ca_file for TLS configuration - logs.file_path for server log ingestion
Extend the DBI pipeline orchestrator to support MySQL instances: - Add translator_mysql.go with dbiMysqlTranslator implementing 4 pipeline types: - metrics/dbi_mysql_N: counter metrics + DBLoad/TopSQL from connectors - logs/dbi_mysql_N: log-to-metrics via count + signaltometrics connectors - logs/dbi_mysql_rawevents_N: raw query sample/top query events - logs/dbi_mysql_serverlogs_N: MySQL server log file ingestion - Add parseDbiMysqlInstances() to parse mysql[] config array - MySQL-specific resource attributes (db.system.name = "mysql") - MySQL-specific log group paths and monitor user exclusion filter - Unit tests for pipeline wiring with single/multiple instances
Add end-to-end config translation test for MySQL DBI: - dbi_mysql_config_linux.conf: input agent config fixture - dbi_mysql_config_linux.json: parsed JSON intermediate fixture - dbi_mysql_config_linux.yaml: golden YAML output (full OTel pipeline) - .mysql_credentials: test passfile for credential loading - Register TestMySQLDbiConfig in tocwconfig_unix_test.go
- Unify the DBI pipeline translator: merge translator_mysql.go into a single dbiTranslator that branches by engine (carried on the instance config), instead of maintaining a separate MySQL translator. - Make component naming consistent across engines using an engine suffix: count/dbi_dbload_<engine>, signaltometrics/dbi_topsql_<engine>, transform/dbi_fix_start_time_<engine>, and per-instance dbi_resource_<engine>_N, dbi_scope_<engine>_N, dbi_logs_<engine>_<dest>_N, dbi_exclude_monitor_<engine>_N. PostgreSQL components now carry the _postgresql suffix for consistency. - Rename engine-specific connector/processor YAMLs for symmetry (dbi_dbload_postgresql.yaml, dbi_topsql_postgresql.yaml, transform_dbi_fix_start_time_postgresql.yaml); the connector and transform translators select the config by engine. - Add the dbi_scope transform processor (cloudwatch.source/solution) to all MySQL pipelines, matching the PostgreSQL implementation. - Use the shared named resourcedetection/opentelemetry processor. - Add the validateOttlSafe check to the unified Translate() so username and instance_name are validated for both engines. - Fix the host metrics process scraper to match the mysqld process (mysqld.*) when MySQL DBI is configured. - Remove TLS/ca_file from the MySQL schema and receiver; MySQL DBI is localhost-only and always connects with TLS disabled. - Fix the MySQL passfile test fixture to the pgpass-style format (host:port:database:username:password) and assert all seven DB Load metrics. - Regenerate golden YAMLs (mysql, postgres, combined configs).
…upport The CloudWatch Console (Cinch) creates separate SSM parameters for each workload configuration (e.g. one for database-insights-postgres and one for database-insights-mysql). When the agent loads multiple config files from the .d/ directory, the config merger needs to deep-merge the database_insights section so that both postgresql and mysql arrays coexist. Without this rule, the agent treats database_insights from two separate config sources as conflicting values and drops one, causing missing log groups in CloudWatch. Register database_insights as a SectionMergeRule under the collect rule, matching the existing pattern for opentelemetry and collect.
- Move the complete 14-metric Top SQL connector config (dbi_topsql_mysql.yaml) into this PR. The full metric set was previously committed in the MySQL workload-detection PR by mistake, which left this PR's embedded config with only 2 metrics while the golden YAML had 14 (mismatch that failed the translation unit tests). All 14 metrics match what the mysqlreceiver publishes on db.server.top_query events. - Remove the user.name attribute from all Top SQL metrics: top queries come from events_statements_summary_by_digest, aggregated by digest and schema, so the receiver does not publish user.name on top_query events (validated against the forked mysqlreceiver RecordDbServerTopQueryEvent). - Remove the duplicate dbiMysqlKey variable in favor of the shared common.DatabaseInsightsMysqlKey. - Regenerate the MySQL golden YAML.
Update the test credential file to use the MySQL-native INI format (Option C) with required host, port, user, and password fields.
Klarsen04
added
the
ready for testing
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
2 times, most recently
from
July 14, 2026 19:10
28af3da to
be7ef57
Compare
Points to amazon-contributing/opentelemetry-collector-contrib PR #577 which replaces pgpass format with .my.cnf INI format for MySQL DBI credential files.
The DB Load connector condition only matched processlist_command ==
"Query", missing prepared statement executions ("Execute") used by
sysbench, ORMs, JDBC, and most real-world applications. This caused
DB Load metrics to undercount active sessions.
Replace the command-based filter with session.status != "idle" to
match all active sessions regardless of command type.
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
from
July 15, 2026 20:01
be7ef57 to
f7367a8
Compare
…rics Add sum_rows_affected, sum_select_range_check, sum_sort_merge_passes, and sum_sort_range to dbi_topsql_mysql.yaml signaltometrics connector config. Top SQL metrics now emit 18 metrics per digest (was 14). Regenerated golden YAML test fixture.
# Conflicts: # go.mod # go.sum # translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_ec2_config.yaml # translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_eks_config.yaml # translator/tocwconfig/sampleConfig/opentelemetry/dbi_postgresql_config_linux.yaml # translator/tocwconfig/tocwconfig_unix_test.go # translator/translate/otel/common/common.go # translator/translate/otel/pipeline/opentelemetry/databaseinsights/translator.go # translator/translate/otel/pipeline/opentelemetry/databaseinsights/translator_test.go # translator/translate/otel/pipeline/opentelemetry/databaseinsights/translators.go # translator/translate/otel/pipeline/opentelemetry/databaseinsights/translators_test.go # translator/translate/otel/pipeline/opentelemetry/hostmetrics/translator_test.go # translator/translate/otel/processor/transformprocessor/translate.go
The mysql DBI translator relied on the receiver's default metric enablement, which leaves mysql.table.size, mysql.replica.sql_delay, and mysql.replica.time_behind_source disabled. Enable them explicitly in the translator (matching the postgres translator pattern) so Database Insights collects table size and replication lag. Regenerated the dbi_mysql golden config accordingly.
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
from
July 23, 2026 15:52
190b830 to
5f2550a
Compare
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
from
July 23, 2026 19:31
c91ca57 to
b944c8a
Compare
Point the mysqlreceiver replace at the DBI metric-additions build and enable the DBI-required MySQL metrics in the translator: - mysql.deadlocks - mysql.connection.count, mysql.connection.errors, mysql.max_used_connections, mysql.commands, mysql.query.count - mysql.history_list_length, mysql.active_transactions - process.threads (DBI hostmetrics process scraper) Regenerated affected golden configs.
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
from
July 23, 2026 22:09
b944c8a to
b3163f4
Compare
Add MySQL database_insights configuration alongside PostgreSQL in the combined sample configs to ensure both engines are represented in multi-feature configs: - Add mysql instance config to combined_v1_v2_ec2_config.json - Add mysql instance config to combined_v1_v2_eks_config.json - Regenerate golden YAML files with all MySQL components: - Receivers: mysql/metrics_0, mysql/events_0, filelog/mysql_0 - Connectors: count/dbi_dbload_mysql, signaltometrics/dbi_topsql_mysql - Processors: filter, transform, scope processors for mysql - Pipelines: logs/dbi_mysql_*, metrics/dbi_mysql_0 - Hostmetrics: process scraper now includes mysqld.* Both combined configs now demonstrate complete dual-engine DBI setups.
…tests The MySQL receiver validates that passfiles have 0600 or 0400 permissions for security. Git doesn't preserve file permissions beyond the executable bit, so checked-out files default to 0644 in CI, causing test failures. Add os.Chmod(0600) calls in TestCombinedV1V2EC2Config and TestCombinedV1V2EKSConfig before config translation to ensure the .mysql_credentials testdata file has the required restrictive permissions. This matches the existing fix in TestDBIMySQLConfigLinux and ensures tests pass reliably in CI and local development environments.
Adjust MySQL query sample configuration to optimize resource usage: - Increase MaxRowsPerQuery from 500 to 5000 (aligns with PostgreSQL) - Decrease MaxQuerySampleCount from 5000 to 1000 (conservative limit) This balances query plan collection with CloudWatch Logs limits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add component translators and pipeline orchestrator for MySQL database monitoring via the OpenTelemetry pipeline, enabling the
opentelemetry.collect.database_insights.mysql[]config section.Continuation of #2176
Changes
Component translators:
Pipeline orchestrator:
Bug fix (new in this PR):
processlist_command == "Query"tosession.status != "idle"to correctly count prepared statement executions (Execute command) used by sysbench, ORMs, and JDBC applicationsTests
make buildandmake lintpass