localhorst/vw-eu-data-act-prometheus-exporter
SHA256
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

fix parsing

Browse Source
This commit is contained in:
localhorst
2026-08-17 10:05:39 +02:00
parent fc60ab416e
commit 69239b8cdf
2 changed files with 111 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+33 -22
View File
@@ -35,35 +35,44 @@ All metrics carry the `vin` label; all names carry the configurable prefix
|---|---|---| |---|---|---|
| `mileage_km` | Gauge | Odometer reading | | `mileage_km` | Gauge | Odometer reading |
| `hvsoc_percent` | Gauge | HV battery state of charge in % | | `hvsoc_percent` | Gauge | HV battery state of charge in % |
| `driver_present` | Gauge (bool) | Driver detected in vehicle | | `driver_present` †| Gauge (bool) | Driver detected in vehicle |
| `cruising_range_km` | Gauge | Remaining range in km | | `cruising_range_km` †| Gauge | Remaining range in km |
| `hvbattery_temperature_max_celsius` | Gauge | Max. HV battery temperature | | `hvbattery_temperature_max_celsius` | Gauge | Max. HV battery temperature |
| `hvbattery_temperature_min_celsius` | Gauge | Min. HV battery temperature | | `hvbattery_temperature_min_celsius` | Gauge | Min. HV battery temperature |
| `charging_state` | Enum | `chargingStatus.currentChargeState` | | `charging_state` | Enum | Current charging state |
| `charge_power_kw` | Gauge | Current charging power in kW | | `charge_power_kw` | Gauge | Current charging power in kW |
| `plug_connection_state` | Enum | Plug connection status | | `plug_connection_state` †| Enum | Plug connection status |
| `target_soc_percent` | Gauge | Charge target in % | | `target_soc_percent` | Gauge | Charge target in % |
| `position_longitude` / `position_latitude` | Gauge | Last known position (only set when **both** coordinates were present in the same dataset) | | `position_longitude` / `position_latitude` †| Gauge | Last known position (only set when **both** coordinates were present in the same dataset) |
| `position_created_timestamp_seconds` | Gauge | Unix timestamp of the last known position | | `position_created_timestamp_seconds` †| Gauge | Unix timestamp of the last known position |
| `locked` | Gauge (bool) | Combined lock status of all doors, trunk and hood (only `unlocked` if at least one component actively reports `UNLOCKED`) | | `locked` | Gauge (bool) | Vehicle locked, as a single combined flag reported directly by the portal |
| `is_parked` | Gauge (bool) | Vehicle parked | | `is_parked` †| Gauge (bool) | Vehicle parked |
| `parking_brake_engaged` | Gauge (bool) | Parking brake engaged | | `parking_brake_engaged` | Gauge (bool) | Parking brake engaged |
| `driving_mode` | Enum | Active driving mode | | `driving_mode` †| Enum | Active driving mode |
| `next_service_type` | Enum | Next due service type | | `next_service_type` †| Enum | Next due service type |
| `service_due_in_days` | Gauge | Remaining days until the next service | | `service_due_in_days` †| Gauge | Remaining days until the next service |
| `last_vehicle_signal_timestamp_seconds` | Gauge | Timestamp of the last vehicle signal (`carCapturedUTCTimestamp`) | | `last_vehicle_signal_timestamp_seconds` | Gauge | Timestamp of the most recent vehicle signal in the latest dataset |
| `uptime_seconds` | Gauge | Uptime of the exporter process | | `uptime_seconds` | Gauge | Uptime of the exporter process |
| `last_successful_scrape_timestamp_seconds` | Gauge | Last scrape with **new** data | | `last_successful_scrape_timestamp_seconds` | Gauge | Last scrape with **new** data |
| `health` | Gauge (bool) | `0` initially and after a failure, `1` once new data has been received at least once | | `health` | Gauge (bool) | `0` initially and after a failure, `1` once new data has been received at least once |
| `http_requests_total` | Counter | Total number of HTTP requests made to the portal (login + API calls) | | `http_requests_total` | Counter | Total number of HTTP requests made to the portal (login + API calls) |
**Note on enum metrics:** The state space (e.g. possible values of † These metrics are defined for API completeness but have no known source
`drivingMode` or `chargingStatus.currentChargeState`) is compiled field in the portal's "continuous" data feed - confirmed absent across 255
best-effort from the sample dataset and comparable projects, and each one real datasets collected via `persist_raw_json` (only
includes an `UNKNOWN` fallback. If an unknown raw value shows up, it is `REPORT_TYPE_ENERGY_CONTENTS`, `REPORT_TYPE_CONFIGURATIONS`,
exported as `UNKNOWN` and additionally logged as `WARNING` - the state list `REPORT_TYPE_CONSUMPTION_VALUES` and `REPORT_TYPE_ADDITIONAL_CONSUMPTION_VALUES`
in `vw-eu-data-act-exporter.py` (the `*_STATES` constants) can then be were ever delivered). They will stay permanently absent from `/metrics`
extended. output with this data source; this is expected, not a bug.
**Note on enum metrics:** `charging_state`'s state list
(`CHARGING_STATE_STATES` in `vw-eu-data-act-exporter.py`) is compiled from
values actually observed in 255 real datasets, plus an `UNKNOWN` fallback.
`driving_mode` and `plug_connection_state` (marked † above) are best-effort
guesses that have never been observed at all. If an unknown raw value shows
up for any enum, it is exported as `UNKNOWN` and additionally logged as
`WARNING` - the state list in `vw-eu-data-act-exporter.py` (the `*_STATES`
constants) can then be extended.
## Installation (systemd) ## Installation (systemd)
@@ -130,9 +139,11 @@ be independent of, and significantly shorter than, `scrape_interval_minutes`
(wrong password) or no datasets available in the portal yet (see (wrong password) or no datasets available in the portal yet (see
prerequisite above). prerequisite above).
- A single metric is missing entirely: the corresponding field has never - A single metric is missing entirely: the corresponding field has never
been present in any dataset fetched so far (e.g. been present in any dataset fetched so far. For the metrics marked †
`position_longitude`/`_latitude`, if both coordinates were never in the table above (e.g. `position_longitude`/`_latitude`,
delivered at the same time). `driving_mode`) this is expected - they simply aren't part of this
data source. For others, use `persist_raw_json` (see Configuration) to
inspect real payloads and check the field name is still correct.
- `WARNING ... Unknown enum value`: a new, not-yet-listed value for an enum - `WARNING ... Unknown enum value`: a new, not-yet-listed value for an enum
metric - the state is exported as `UNKNOWN`, extend the list in the metric - the state is exported as `UNKNOWN`, extend the list in the
source code if needed. source code if needed.
vw-eu-data-act-exporter.py
+78 -77
View File
@@ -306,6 +306,33 @@ def flatten_dataset(record: dict) -> dict[str, str]:
return fields return fields
# A real dataset's Data array bundles several sub-reports concatenated
# together, so these two field names each occur multiple times per dataset
# with unrelated values (confirmed via their distinct per-entry "key"
# UUIDs) - flatten_dataset()'s last-value-wins is unsafe for them.
SIGNAL_TIMESTAMP_FIELDS = ("car_captured_time", "car_captured_utc_timestamp")
def latest_signal_timestamp_unix(record: dict) -> float | None:
"""Returns the most recent unix timestamp found under any of
SIGNAL_TIMESTAMP_FIELDS in the raw Data array, or ``None`` if none are
present/parseable."""
best = None
for entry in record.get("Data", []):
if entry.get("dataFieldName") not in SIGNAL_TIMESTAMP_FIELDS:
continue
raw = entry.get("value")
if not raw:
continue
try:
unix = parse_iso8601_to_unix(raw)
except (TypeError, ValueError):
continue
if best is None or unix > best:
best = unix
return best
def persist_raw_dataset(vin: str, record: dict, *, suffix: str = "") -> None: def persist_raw_dataset(vin: str, record: dict, *, suffix: str = "") -> None:
"""Writes the JSON extracted from a downloaded dataset to disk, so it """Writes the JSON extracted from a downloaded dataset to disk, so it
can be inspected later to refine the field mapping below. Only the can be inspected later to refine the field mapping below. Only the
@@ -320,11 +347,12 @@ def persist_raw_dataset(vin: str, record: dict, *, suffix: str = "") -> None:
log.debug("Persisted raw dataset for VIN %s to %s", vin, dest) log.debug("Persisted raw dataset for VIN %s to %s", vin, dest)
def fetch_latest_dataset_fields(vin: str) -> dict[str, str] | None: def fetch_latest_dataset_fields(vin: str) -> dict[str, str | float | None] | None:
"""Logs in, downloads the latest dataset for *vin* and returns the """Logs in, downloads the latest dataset for *vin* and returns the
flattened fields. Returns ``None`` if no datasets are (yet) available - flattened fields, plus a synthetic "_signal_timestamp_unix" key (see
this is explicitly NOT a failure. Raises an exception on login/HTTP/parse latest_signal_timestamp_unix()). Returns ``None`` if no datasets are
errors (= failure).""" (yet) available - this is explicitly NOT a failure. Raises an exception
on login/HTTP/parse errors (= failure)."""
session = requests.Session() session = requests.Session()
session.headers.update({"User-Agent": random.choice(config.user_agents)}) session.headers.update({"User-Agent": random.choice(config.user_agents)})
@@ -385,7 +413,9 @@ def fetch_latest_dataset_fields(vin: str) -> dict[str, str] | None:
except OSError as exc: except OSError as exc:
log.warning("Could not persist raw dataset for VIN %s: %s", vin, exc) log.warning("Could not persist raw dataset for VIN %s: %s", vin, exc)
return flatten_dataset(record) fields = flatten_dataset(record)
fields["_signal_timestamp_unix"] = latest_signal_timestamp_unix(record)
return fields
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -454,7 +484,7 @@ g_position_created_timestamp_seconds = Gauge(
) )
g_locked = Gauge( g_locked = Gauge(
f"{P}locked", f"{P}locked",
"Combined lock status of all doors, trunk and hood (1=locked)", "Vehicle locked (1=locked), as a single combined flag from the portal",
["vin"], ["vin"],
) )
g_is_parked = Gauge(f"{P}is_parked", "Vehicle parked (1=yes)", ["vin"]) g_is_parked = Gauge(f"{P}is_parked", "Vehicle parked (1=yes)", ["vin"])
@@ -466,7 +496,8 @@ g_service_due_in_days = Gauge(
) )
g_last_vehicle_signal_timestamp_seconds = Gauge( g_last_vehicle_signal_timestamp_seconds = Gauge(
f"{P}last_vehicle_signal_timestamp_seconds", f"{P}last_vehicle_signal_timestamp_seconds",
"Unix timestamp of the last vehicle signal (carCapturedUTCTimestamp)", "Unix timestamp of the most recent vehicle signal in the latest dataset "
"(max of all car_captured_time/car_captured_utc_timestamp entries)",
["vin"], ["vin"],
) )
@@ -487,18 +518,19 @@ c_http_requests_total = Counter(
["vin"], ["vin"],
) )
# Enum metrics. The state space is best-effort (compiled from the sample # Enum metrics. driving_mode/plug_connection_state/next_service_type are not
# dataset and comparable projects) plus an "UNKNOWN" fallback for unknown raw # delivered by the "continuous" data feed at all (confirmed against 255 real
# values - these are additionally logged as WARNING so the list can be # datasets - see README) and are kept only for API completeness; they will
# extended if needed. # never actually populate. charging_state's states below are the values
# actually observed in those 255 real datasets, plus an "UNKNOWN" fallback
# for anything new - unknown raw values are additionally logged as WARNING
# so the list can be extended if needed.
DRIVING_MODE_STATES = ["standard", "eco", "comfort", "sport", "individual", "UNKNOWN"] DRIVING_MODE_STATES = ["standard", "eco", "comfort", "sport", "individual", "UNKNOWN"]
CHARGING_STATE_STATES = [ CHARGING_STATE_STATES = [
"OFF", "CHARGE_STATE_NOT_READY_FOR_CHARGING",
"READY_FOR_CHARGING", "CHARGE_STATE_READY_FOR_CHARGING",
"NOT_READY_FOR_CHARGING", "CHARGE_STATE_CHARGING_HV_BATTERY",
"CHARGING", "CHARGE_STATE_CHARGE_PURPOSE_REACHED_AND_NOT_CONSERVATION_CHARGING",
"CONSERVING",
"ERROR",
"UNKNOWN", "UNKNOWN",
] ]
PLUG_CONNECTION_STATES = ["CONNECTED", "DISCONNECTED", "UNKNOWN"] PLUG_CONNECTION_STATES = ["CONNECTED", "DISCONNECTED", "UNKNOWN"]
@@ -532,59 +564,38 @@ e_service_type = Enum(
states=SERVICE_TYPE_STATES, states=SERVICE_TYPE_STATES,
) )
# Field name (dataFieldName in the dataset) -> (Gauge, conversion function) # Field name (dataFieldName in the dataset) -> (Gauge, conversion function).
# Verified against 255 real datasets in raw_json/: each of these fields has
# exactly one distinct originating "key" per file (mileage.value has two,
# but they agreed on value in 0/242 sampled disagreements) - i.e. they are
# genuinely unambiguous single-valued fields, unlike e.g. "timestamp" or
# "car_captured_time" which bundle multiple unrelated sub-reports under the
# same dataFieldName (see latest_signal_timestamp_unix() above).
#
# driver_present, cruising_range_km, position_longitude/_latitude/_created,
# is_parked, service_due_in_days and the driving_mode/plug_connection_state/
# next_service_type enums have no known source field at all in this data
# feed (see README) and are intentionally left unmapped below - their Gauge/
# Enum objects stay defined but will simply never be set.
GAUGE_FIELD_MAP: list[tuple[str, Gauge, callable]] = [ GAUGE_FIELD_MAP: list[tuple[str, Gauge, callable]] = [
("mileage_info.value", g_mileage_km, parse_float), ("mileage.value", g_mileage_km, parse_float),
("hvsoc_info.value", g_hvsoc_percent, parse_float), ("battery_state_report.soc", g_hvsoc_percent, parse_float),
("Driver Presence", g_driver_present, parse_bool_as_float), ("max_temperature", g_hvbattery_temp_max_celsius, parse_float),
("batteryStatus.cruisingRange.range", g_cruising_range_km, parse_float), ("min_temperature", g_hvbattery_temp_min_celsius, parse_float),
( ("battery_state_report.charge_power", g_charge_power_kw, parse_float),
"hvbatterytemperature_info.max_temperature.value", ("settings.target_soc", g_target_soc_percent, parse_float),
g_hvbattery_temp_max_celsius, ("parking_brake", g_parking_brake_engaged, parse_bool_as_float),
parse_float, ("locked", g_locked, parse_bool_as_float),
),
(
"hvbatterytemperature_info.min_temperature.value",
g_hvbattery_temp_min_celsius,
parse_float,
),
("chargingStatus.chargePower_kW", g_charge_power_kw, parse_float),
("targetSoc_pct", g_target_soc_percent, parse_float),
("positionCreated", g_position_created_timestamp_seconds, parse_iso8601_to_unix),
("isParked", g_is_parked, parse_bool_as_float),
("parking_brake_info.value", g_parking_brake_engaged, parse_bool_as_float),
("service_maintenance_info.due_in_time.value", g_service_due_in_days, parse_float),
(
"carCapturedUTCTimestamp",
g_last_vehicle_signal_timestamp_seconds,
parse_iso8601_to_unix,
),
] ]
ENUM_FIELD_MAP: list[tuple[str, Enum, list[str]]] = [ ENUM_FIELD_MAP: list[tuple[str, Enum, list[str]]] = [
("drivingMode", e_driving_mode, DRIVING_MODE_STATES),
("chargingStatus.currentChargeState", e_charging_state, CHARGING_STATE_STATES),
( (
"plugStatusItem.plugConnectionState", "charging_state_report.current_charge_state",
e_plug_connection_state, e_charging_state,
PLUG_CONNECTION_STATES, CHARGING_STATE_STATES,
), ),
("service_maintenance_info.service_type", e_service_type, SERVICE_TYPE_STATES),
] ]
# Lock components -> internal name. If a component is missing from a dataset
# entirely (even across all scrapes so far), it is treated as "LOCKED" (safe
# default), so a single door actively reporting locked does not on its own
# report "vehicle locked" if another component is still unknown.
LOCK_FIELDS = {
"door_info.front_left.door_lock_status.value": "front_left",
"door_info.front_right.door_lock_status.value": "front_right",
"door_info.rear_left.door_lock_status.value": "rear_left",
"door_info.rear_right.door_lock_status.value": "rear_right",
"trunk_lid_info.trunk_lid_lock_status.value": "trunk",
"hood_info.hood_lock_status.value": "hood",
}
class UptimeCollector: class UptimeCollector:
"""Computes uptime freshly on every scrape instead of maintaining a """Computes uptime freshly on every scrape instead of maintaining a
@@ -609,10 +620,7 @@ class UptimeCollector:
class VehiclePoller: class VehiclePoller:
def __init__(self, vin: str) -> None: def __init__(self, vin: str) -> None:
self.vin = vin self.vin = vin
self.lock_components: dict[str, str] = { self.last_signal_unix: float | None = None
name: "LOCKED" for name in LOCK_FIELDS.values()
}
self.last_signal_ts_raw: str | None = None
def apply_fields(self, fields: dict[str, str]) -> None: def apply_fields(self, fields: dict[str, str]) -> None:
vin = self.vin vin = self.vin
@@ -634,14 +642,6 @@ class VehiclePoller:
log.warning("Unknown enum value for %s (VIN %s): %r", key, vin, raw) log.warning("Unknown enum value for %s (VIN %s): %r", key, vin, raw)
enum_metric.labels(vin=vin).state(state) enum_metric.labels(vin=vin).state(state)
for key, comp_name in LOCK_FIELDS.items():
if key in fields:
self.lock_components[comp_name] = str(fields[key])
locked = (
0.0 if any(v == "UNLOCKED" for v in self.lock_components.values()) else 1.0
)
g_locked.labels(vin=vin).set(locked)
if "longitude" in fields and "latitude" in fields: if "longitude" in fields and "latitude" in fields:
try: try:
g_position_longitude.labels(vin=vin).set( g_position_longitude.labels(vin=vin).set(
@@ -651,9 +651,10 @@ class VehiclePoller:
except (TypeError, ValueError) as exc: except (TypeError, ValueError) as exc:
log.warning("Could not parse position (VIN %s): %s", vin, exc) log.warning("Could not parse position (VIN %s): %s", vin, exc)
new_ts = fields.get("carCapturedUTCTimestamp") new_signal_unix = fields.get("_signal_timestamp_unix")
if new_ts is not None and new_ts != self.last_signal_ts_raw: if new_signal_unix is not None and new_signal_unix != self.last_signal_unix:
self.last_signal_ts_raw = new_ts self.last_signal_unix = new_signal_unix
g_last_vehicle_signal_timestamp_seconds.labels(vin=vin).set(new_signal_unix)
g_last_successful_scrape_timestamp_seconds.labels(vin=vin).set(time.time()) g_last_successful_scrape_timestamp_seconds.labels(vin=vin).set(time.time())
g_health.labels(vin=vin).set(1.0) g_health.labels(vin=vin).set(1.0)