goober/nte-exporter
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Decode reward keys and use mapping files

Browse source 
Add decoding for Monopoly reward keys and switch reward metadata to be looked up by decoded id. Implement decode_reward_key() and infer_reward_type(); replace the previous raw-key KNOWN_REWARDS approach with REWARDS_BY_ID built from mappings/*.json. Add mappings/items.json and expand mappings/arcs.json. Update decoder to produce reward_id, use REWARDS_BY_ID, and change guess_quantity() to accept reward_id. Refactor pool_mappings to reuse the new mapping loader, and update tests and docs to validate mapping files and document the reward key encoding/behavior.
This commit is contained in:
Golumpa 2026-06-11 11:09:02 +01:00
parent 401df44ecd
commit 354b0e9232
8 changed files with 197 additions and 113 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/limitations.md
View file

@ -12,6 +12,6 @@ Known limitations:
- Live capture currently uses Windows raw sockets and requires administrator permission.
- The file adapter reads mitmproxy `.flows` captures for research and testing.
- The more stable and reliable Npcap/libpcap capture is not implemented yet.
- Reward mappings are research mappings and should be expanded as more samples are decoded.
- Reward keys decode to their reward id string, so unknown rewards still export a usable `reward_id`; display names/ranks come from the mapping JSON files and should be expanded as new rewards appear.
Privacy guardrail: raw packet data must not be included in sanitized exports.

10
docs/packet-format.md
View file

@ -22,7 +22,15 @@ Decoded fields:
- `source_flag = -4` means Chase Reward
- Timestamp is an 8-byte little-endian value
- `unix_seconds = little_endian_u64(timestamp_raw) / 40000000 - 62135596800`
- Rewards are identified by stable binary keys
- Reward keys are the reward id string encoded one character per byte as
`ASCII * 4` with carry chaining into the next byte. The final byte is the
pending carry (`00` or `01`) acting as a terminator, or is omitted.
Examples: `98bdc9ad7dd9a5b99501` decodes to `fork_vine`,
`10a58d9539bdc9b585b101` to `DiceNormal`, `c4c0cccc00` to character id `1033`.
Arc/Gashapon history uses the same scheme at `ASCII * 2`.
- The decoder decodes the key to its id string and looks up display metadata
in `mappings/arcs.json`, `mappings/characters.json`, and `mappings/items.json`.
Unknown rewards still export their decoded id with empty name/rank.
Page and row numbers are research metadata only. They must not be used for permanent dedupe because they shift when new history appears.

35
mappings/arcs.json
View file

@ -7,8 +7,41 @@
"fork_PaperPlane": {"name": "Clear Skies", "rank": "A"},
"fork_mofeikesi": {"name": "Good Boy's Grand Adventure", "rank": "S"},
"fork_wuhuakuang": {"name": "The Forgotten", "rank": "A"},
"fork_whuakuang": {"name": "The Forgotten", "rank": "A"},
"fork_jingmotingyuan": {"name": "Camellia Society", "rank": "S"},
"fork_Kite": {"name": "Watch Your Heads!", "rank": "A"},
"fork_yuren": {"name": "Umbrella", "rank": "A"},
"fork_yaodao": {"name": "Drawn Blade", "rank": "A"}
"fork_yaodao": {"name": "Drawn Blade", "rank": "A"},
"fork_Arachne": {"name": "Eternal Waltz", "rank": "S"},
"fork_BitGame": {"name": "Blow up the Crowd", "rank": "S"},
"fork_BitterCake": {"name": "The Good, The Bad, The Bitter", "rank": "A"},
"fork_BlackBook": {"name": "Youthful Fantasy", "rank": "S"},
"fork_Baozhatang": {"name": "Fluff of Fearlessness", "rank": "S"},
"fork_bopu": {"name": "Cosmos Daze, Wild Reverie", "rank": "A"},
"fork_Quanjitang": {"name": "Fluff of Fortitude", "rank": "S"},
"fork_Butterfly": {"name": "Reality Refuge", "rank": "S"},
"fork_Castle": {"name": "Call of the Twisted City", "rank": "A"},
"fork_Qiaoqiao": {"name": "Time Bandit", "rank": "A"},
"fork_jiaojuan": {"name": "Shiny Days", "rank": "A"},
"fork_KnightCandy": {"name": "Fluff of Ferocity", "rank": "S"},
"fork_koinobori": {"name": "A Time Will Come", "rank": "A"},
"fork_lingganzhongjiezhe": {"name": "Mind Royale", "rank": "A"},
"fork_mamen": {"name": "Contemplative Cat", "rank": "S"},
"fork_moon": {"name": "Stellar Veil", "rank": "S"},
"fork_MotorCandy": {"name": "Fluff of Fleetness", "rank": "S"},
"fork_Nakupeda": {"name": "Your Happiness is Priceless", "rank": "S"},
"fork_NestBird": {"name": "Tears Beneath the Mask", "rank": "S"},
"fork_oulaquantao": {"name": "Oraora!", "rank": "A"},
"fork_PoliceRat": {"name": "Hethereau's Keeper", "rank": "S"},
"fork_rishi": {"name": "Day Off", "rank": "S"},
"fork_Rose": {"name": "The Last Rose", "rank": "S"},
"fork_snowman": {"name": "The Fools' Spring", "rank": "A"},
"fork_spider": {"name": "Failing You, Heavy in My Heart", "rank": "A"},
"fork_ThiefCandy": {"name": "Fluff of Finesse", "rank": "S"},
"fork_TigerTally": {"name": "Ready-Ready", "rank": "S"},
"fork_Time": {"name": "Marching Beyond Time", "rank": "S"},
"fork_tuansanlang": {"name": "The Great Thief", "rank": "A"},
"fork_Whale": {"name": "Song of the Whale", "rank": "S"},
"fork_worldrain": {"name": "The Rain That Shook the World", "rank": "S"},
"fork_Wushoutieyu": {"name": "Raging Flames", "rank": "S"}
}

8
mappings/items.json Normal file
View file

@ -0,0 +1,8 @@
{
"DiceNormal": {"type": "item", "name": "Fabricated Dice", "rank": null},
"Dice_ticket_01": {"type": "item", "name": "Warp Piece", "rank": null},
"Dice_ticket_02": {"type": "item", "name": "Lost Piece", "rank": null},
"Dicelimite": {"type": "item", "name": "Solid Dice", "rank": null},
"Fashion_vehicle_1010_V008": {"type": "cosmetic", "name": "Fashion Vehicle Skin 1010 V008", "rank": null},
"Fashion_Glide_1010": {"type": "cosmetic", "name": "Fashion Glider 1010", "rank": null}
}

66
src/nte_history_exporter/decoder/protocol.py
View file

@ -15,7 +15,52 @@ from nte_history_exporter.constants import (
TIMESTAMP_TICKS_PER_SECOND,
VALID_DICE_FIELDS,
)
from nte_history_exporter.mappings import KNOWN_REWARDS
from nte_history_exporter.mappings import REWARDS_BY_ID
REWARD_ID_CHARS = frozenset("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_")
def decode_reward_key(raw: bytes) -> str:
"""Decode a Monopoly reward key into its reward id string.
Keys encode the id one character per byte as ASCII*4 with carry chaining
(Arc history uses the same scheme at ASCII*2). The final byte is either the
pending carry (0 or 1) acting as a terminator, or a regular character byte.
The carry bit makes some bytes ambiguous, so decode with backtracking and
accept only parses made of identifier characters.
"""
results: list[str] = []
def walk(index: int, carry: int, acc: str) -> None:
if results:
return
if index == len(raw):
if carry == 0:
results.append(acc)
return
byte = raw[index]
if index == len(raw) - 1 and byte == carry:
results.append(acc)
return
for carry_out in (0, 1):
value = byte - carry + 256 * carry_out
if value % 4 == 0 and chr(value // 4) in REWARD_ID_CHARS:
walk(index + 1, carry_out, acc + chr(value // 4))
walk(0, 0, "")
return results[0] if results else ""
def infer_reward_type(reward_id: str) -> str:
if not reward_id:
return ""
if reward_id.startswith("fork_"):
return "arc"
if reward_id.isdigit():
return "character"
if reward_id.startswith("Fashion_"):
return "cosmetic"
return "item"
def decode_history_timestamp(raw8: bytes) -> tuple[int, float, str]:
@ -98,20 +143,20 @@ def classify_result_type(
return "dice", None
def guess_quantity(chunk_hex: str, key_hex: str, result_type: str | None = None) -> int | None:
if key_hex == "10a58d957dd1a58dad95d17dc1c400":
def guess_quantity(chunk_hex: str, reward_id: str, result_type: str | None = None) -> int | None:
if reward_id == "Dice_ticket_01":
if result_type == "chase_reward":
return 30
return 4
if key_hex == "10a58d9539bdc9b585b101":
if reward_id == "DiceNormal":
return 1
if key_hex == "10a58d957dd1a58dad95d17dc1c800":
if reward_id == "Dice_ticket_02":
if "c8b0d4c0" in chunk_hex:
return 50
if "c8b0ccc0" in chunk_hex:
return 30
return None
if key_hex:
if reward_id:
return 1
return None
@ -142,7 +187,8 @@ def decode_response_records(response_content: bytes) -> list[dict[str, Any]]:
dice = -4
dice_raw = -4
key_hex = extract_key(chunk)
reward = KNOWN_REWARDS.get(key_hex, {})
reward_id = decode_reward_key(bytes.fromhex(key_hex)) if key_hex else ""
reward = REWARDS_BY_ID.get(reward_id, {})
timestamp_raw = response_content[marker_offset + len(marker) : marker_offset + len(marker) + 8]
timestamp_ticks, timestamp_unix, timestamp_decoded = decode_history_timestamp(timestamp_raw)
chunk_hex = chunk.hex()
@ -163,11 +209,11 @@ def decode_response_records(response_content: bytes) -> list[dict[str, Any]]:
"dice_raw_u32": dice_raw,
"dice_offset_in_record": dice_offset,
"reward_key_hex": key_hex,
"reward_type": reward.get("type", ""),
"reward_id": reward.get("id", ""),
"reward_type": reward.get("type") or infer_reward_type(reward_id),
"reward_id": reward_id,
"reward_name": reward.get("name", ""),
"reward_rank": reward.get("rank"),
"quantity": guess_quantity(chunk_hex, key_hex, result_type),
"quantity": guess_quantity(chunk_hex, reward_id, result_type),
"timestamp_raw_hex": timestamp_raw.hex(),
"timestamp_ticks": timestamp_ticks,
"timestamp_unix": f"{timestamp_unix:.6f}",

109
src/nte_history_exporter/mappings.py
View file

@ -1,92 +1,33 @@
KNOWN_REWARDS = {
"98bdc9ad7d91d5cdd189a5b901": {"type": "arc", "id": "fork_dustbin", "name": "Dangerous Game", "rank": "B"},
"98bdc9ad7dd9a5b99501": {"type": "arc", "id": "fork_vine", "name": "Be Happy", "rank": "B"},
"98bdc9ad7db9bdb9bdcd01": {"type": "arc", "id": "fork_nonos", "name": "First Step to Success", "rank": "B"},
"98bdc9ad7d85c1c1b1a585b98d9501": {"type": "arc", "id": "fork_appliance", "name": "\"Real Music\"", "rank": "B"},
"98bdc9ad7d41c9bdad85c9e5bdb901": {"type": "arc", "id": "fork_Prokaryon", "name": "Us.", "rank": "B"},
"98bdc9ad7d4185c195c941b185b99501": {"type": "arc", "id": "fork_PaperPlane", "name": "Clear Skies", "rank": "A"},
"98bdc9ad7ddda1d585add585b99d01": {"type": "arc", "id": "fork_wuhuakuang", "name": "The Forgotten", "rank": "A"},
"98bdc9ad7dddd5a1d585add585b99d01": {"type": "arc", "id": "fork_wuhuakuang", "name": "The Forgotten", "rank": "A"},
"98bdc9ad7d2da5d19501": {"type": "arc", "id": "fork_Kite", "name": "Watch Your Heads!", "rank": "A"},
"98bdc9ad7de5d5c995b901": {"type": "arc", "id": "fork_yuren", "name": "Umbrella", "rank": "A"},
"98bdc9ad7de585bd9185bd01": {"type": "arc", "id": "fork_yaodao", "name": "Drawn Blade", "rank": "A"},
"10a58d9539bdc9b585b101": {"type": "item", "id": "DiceNormal", "name": "Fabricated Dice", "rank": None},
"10a58d957dd1a58dad95d17dc1c400": {"type": "item", "id": "Dice_ticket_01", "name": "Warp Piece", "rank": None},
"10a58d957dd1a58dad95d17dc1c800": {"type": "item", "id": "Dice_ticket_02", "name": "Lost Piece", "rank": None},
"10a58d95b1a5b5a5d19501": {"type": "item", "id": "Dicelimite", "name": "Solid Dice", "rank": None},
"1885cda1a5bdb97dd995a1a58db1957dc5c0c4c07c59c1c0e000": {
"type": "cosmetic",
"id": "Fashion_vehicle_1010_V008",
"name": "Fashion Vehicle Skin 1010 V008",
"rank": None,
},
"1885cda1a5bdb97d1db1a591957dc5c0c4c000": {
"type": "cosmetic",
"id": "Fashion_glide_1010",
"name": "Fashion Glider 1010",
"rank": None,
},
"c4c0cccc00": {"type": "character", "id": "1033", "name": "Adler", "rank": "A"},
"c4c0c8c000": {"type": "character", "id": "1020", "name": "Haniel", "rank": "A"},
"c4c0c8cc00": {"type": "character", "id": "1023", "name": "Baicang", "rank": "S"},
"c4c0c8d400": {"type": "character", "id": "1025", "name": "Hathor", "rank": "S"},
"c4c0c4e400": {"type": "character", "id": "1019", "name": "Mint", "rank": "A"},
"c4c0c0e000": {"type": "character", "id": "1008", "name": "Skia", "rank": "A"},
"c4c0dcc0": {"type": "character", "id": "1070", "name": "Aurelia", "rank": "A"},
"c4c0dcc000": {"type": "character", "id": "1070", "name": "Aurelia", "rank": "A"},
"c4c0c8c4": {"type": "character", "id": "1021", "name": "Edgar", "rank": "A"},
"c4c0c8c400": {"type": "character", "id": "1021", "name": "Edgar", "rank": "A"},
}
from __future__ import annotations
ARC_META = {
"fork_nonos": {"name": "First Step to Success", "rank": "B"},
"fork_Prokaryon": {"name": "Us.", "rank": "B"},
"fork_appliance": {"name": "\"Real Music\"", "rank": "B"},
"fork_PaperPlane": {"name": "Clear Skies", "rank": "A"},
"fork_mofeikesi": {"name": "Good Boy's Grand Adventure", "rank": "S"},
"fork_dustbin": {"name": "Dangerous Game", "rank": "B"},
"fork_vine": {"name": "Be Happy", "rank": "B"},
"fork_Kite": {"name": "Watch Your Heads!", "rank": "A"},
"fork_wuhuakuang": {"name": "The Forgotten", "rank": "A"},
"fork_jingmotingyuan": {"name": "Camellia Society", "rank": "S"},
"fork_yuren": {"name": "Umbrella", "rank": "A"},
"fork_yaodao": {"name": "Drawn Blade", "rank": "A"},
}
import json
from pathlib import Path
from typing import Any
CHARACTERS = {
"1003": {"name": "Sakiri", "rank": "S"},
"1004": {"name": "Lacrimosa", "rank": "S"},
"1008": {"name": "Skia", "rank": "A"},
"1010": {"name": "Nanally", "rank": "S"},
"1019": {"name": "Mint", "rank": "A"},
"1020": {"name": "Haniel", "rank": "A"},
"1021": {"name": "Edgar", "rank": "A"},
"1023": {"name": "Baicang", "rank": "S"},
"1025": {"name": "Hathor", "rank": "S"},
"1033": {"name": "Adler", "rank": "A"},
"1039": {"name": "Fadia", "rank": "S"},
"1046": {"name": "Zero", "rank": "S"},
"1051": {"name": "Zero", "rank": "S"},
"1052": {"name": "Hotori", "rank": "S"},
"1054": {"name": "Daffodill", "rank": "S"},
"1055": {"name": "Jiuyuan", "rank": "S"},
"1056": {"name": "Lacrimosa", "rank": "S"},
"1070": {"name": "Aurelia", "rank": "A"},
"1073": {"name": "Chiz", "rank": "S"},
}
PROJECT_ROOT = Path(__file__).resolve().parents[2]
MAPPINGS_DIR = PROJECT_ROOT / "mappings"
def character_id_to_key(character_id: str) -> str:
return "".join(f"{0xC0 + int(digit) * 4:02x}" for digit in character_id) + "00"
def load_mapping_file(filename: str) -> dict[str, Any]:
return json.loads((MAPPINGS_DIR / filename).read_text(encoding="utf-8"))
for _character_id, _info in CHARACTERS.items():
KNOWN_REWARDS.setdefault(
character_id_to_key(_character_id),
{
ARC_META: dict[str, dict[str, Any]] = load_mapping_file("arcs.json")
CHARACTERS: dict[str, dict[str, Any]] = load_mapping_file("characters.json")
ITEMS: dict[str, dict[str, Any]] = load_mapping_file("items.json")
# Reward keys in Monopoly history packets are the reward id string itself
# (ASCII*4 with carry, see decoder.protocol.decode_reward_key), so all reward
# metadata is looked up by decoded id rather than by raw key bytes.
REWARDS_BY_ID: dict[str, dict[str, Any]] = {}
for _arc_id, _meta in ARC_META.items():
REWARDS_BY_ID[_arc_id] = {"type": "arc", "id": _arc_id, "name": _meta["name"], "rank": _meta["rank"]}
for _character_id, _meta in CHARACTERS.items():
REWARDS_BY_ID[_character_id] = {
"type": "character",
"id": _character_id,
"name": _info["name"],
"rank": _info["rank"],
},
)
"name": _meta["name"],
"rank": _meta["rank"],
}
for _item_id, _meta in ITEMS.items():
REWARDS_BY_ID[_item_id] = {"type": _meta["type"], "id": _item_id, "name": _meta["name"], "rank": _meta.get("rank")}

8
src/nte_history_exporter/pool_mappings.py
View file

@ -1,11 +1,8 @@
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
PROJECT_ROOT = Path(__file__).resolve().parents[2]
MAPPINGS_DIR = PROJECT_ROOT / "mappings"
from nte_history_exporter.mappings import load_mapping_file
POOL_MAPPING_FILES = {
"permanent": "permanent_board.json",
@ -15,8 +12,7 @@ POOL_MAPPING_FILES = {
def load_pool_mapping(pool_key: str) -> dict[str, Any]:
filename = POOL_MAPPING_FILES[pool_key]
return json.loads((MAPPINGS_DIR / filename).read_text(encoding="utf-8"))
return load_mapping_file(POOL_MAPPING_FILES[pool_key])
def load_pool_mappings() -> dict[str, dict[str, Any]]:

68
tests/test_boundary_export.py
View file

@ -17,7 +17,8 @@ from nte_history_exporter.constants import LIMITED_CHARACTER_MARKER, MARKER
from nte_history_exporter.decoder.boundary import page_gap_warnings
from nte_history_exporter.decoder.protocol import decode_response_records, history_request_kind
from nte_history_exporter.constants import POOL_META
from nte_history_exporter.mappings import ARC_META, CHARACTERS
from nte_history_exporter.mappings import ARC_META, CHARACTERS, ITEMS, REWARDS_BY_ID
from nte_history_exporter.decoder.protocol import decode_reward_key, infer_reward_type
from nte_history_exporter.decoder.arc import (
build_arc_rows_from_pairs,
decode_arc_key,
@ -32,6 +33,8 @@ from nte_history_exporter.pool_mappings import load_pool_mappings, pool_meta_fro
def load_reference_csv(name):
path = EXPORTS / name
if not path.exists():
path = EXPORTS / "old" / name
if not path.exists():
raise unittest.SkipTest(f"private reference fixture not present: {path}")
rows = []
@ -88,14 +91,61 @@ class BoundaryExportTests(unittest.TestCase):
with self.subTest(pool_key=pool_key):
self.assertEqual(pool_meta_from_mapping(mapping), POOL_META[pool_key])
def test_reward_mapping_json_matches_runtime_mappings(self):
with (ROOT / "mappings" / "arcs.json").open(encoding="utf-8") as f:
arc_json = json.load(f)
with (ROOT / "mappings" / "characters.json").open(encoding="utf-8") as f:
character_json = json.load(f)
def test_reward_mapping_files_have_expected_shape(self):
self.assertTrue(ARC_META)
for arc_id, meta in ARC_META.items():
with self.subTest(arc_id=arc_id):
self.assertTrue(arc_id.startswith("fork_"))
self.assertIn("name", meta)
self.assertIn(meta.get("rank"), ("S", "A", "B"))
self.assertEqual(arc_json, ARC_META)
self.assertEqual(character_json, CHARACTERS)
self.assertTrue(CHARACTERS)
for character_id, info in CHARACTERS.items():
with self.subTest(character_id=character_id):
self.assertTrue(character_id.isdigit())
self.assertIn("name", info)
self.assertIn(info.get("rank"), ("S", "A"))
self.assertTrue(ITEMS)
for item_id, info in ITEMS.items():
with self.subTest(item_id=item_id):
self.assertIn(info.get("type"), ("item", "cosmetic"))
self.assertIn("name", info)
def test_rewards_by_id_merges_all_mapping_files(self):
for reward_id in (*ARC_META, *CHARACTERS, *ITEMS):
with self.subTest(reward_id=reward_id):
reward = REWARDS_BY_ID[reward_id]
self.assertEqual(reward["id"], reward_id)
self.assertIn(reward["type"], ("arc", "character", "item", "cosmetic"))
def test_decode_reward_key_round_trips_observed_keys(self):
observed = {
"98bdc9ad7dd9a5b99501": "fork_vine",
"98bdc9ad7d41c9bdad85c9e5bdb901": "fork_Prokaryon",
"98bdc9ad7ddda1d585add585b99d01": "fork_whuakuang",
"98bdc9ad7dddd5a1d585add585b99d01": "fork_wuhuakuang",
"10a58d9539bdc9b585b101": "DiceNormal",
"10a58d957dd1a58dad95d17dc1c400": "Dice_ticket_01",
"10a58d957dd1a58dad95d17dc1c800": "Dice_ticket_02",
"10a58d95b1a5b5a5d19501": "Dicelimite",
"1885cda1a5bdb97d1db1a591957dc5c0c4c000": "Fashion_Glide_1010",
"1885cda1a5bdb97dd995a1a58db1957dc5c0c4c07c59c1c0e000": "Fashion_vehicle_1010_V008",
"c4c0cccc00": "1033",
"c4c0dcc000": "1070",
"c4c0dcc0": "1070",
"c4c0c8c4": "1021",
}
for key_hex, expected_id in observed.items():
with self.subTest(key_hex=key_hex):
self.assertEqual(decode_reward_key(bytes.fromhex(key_hex)), expected_id)
def test_infer_reward_type_for_unmapped_ids(self):
self.assertEqual(infer_reward_type("fork_newarc"), "arc")
self.assertEqual(infer_reward_type("1099"), "character")
self.assertEqual(infer_reward_type("Fashion_hat_2000"), "cosmetic")
self.assertEqual(infer_reward_type("Dice_ticket_03"), "item")
self.assertEqual(infer_reward_type(""), "")
def test_uid_source_matches_v4_reference(self):
rows = load_reference_csv("monopoly_history_poc_10_all_44_pages_v4.csv")
@ -133,6 +183,8 @@ class BoundaryExportTests(unittest.TestCase):
exported = [row for row in annotated if row["export_record"] is True]
json_path = EXPORTS / "monopoly_history_export_10_all_44_pages_v4.json"
if not json_path.exists():
json_path = EXPORTS / "old" / "monopoly_history_export_10_all_44_pages_v4.json"
if not json_path.exists():
raise unittest.SkipTest(f"private reference fixture not present: {json_path}")
with json_path.open(encoding="utf-8") as f: