Added Mystery Box Support

Added the event specific Mystery Box support to the exporter. Closes #7
This commit is contained in:
Golumpa 2026-07-16 00:50:14 +01:00
parent 2b33ea63af
commit 9eae9b4a15
24 changed files with 627 additions and 19 deletions

View file

@ -19,10 +19,11 @@ Prototype CLI exporter for **Neverness to Everness** pull history — decodes yo
| Monopoly | Standard Board | `Lottery_Permanent` | Per-banner |
| Monopoly | Limited Character Board | `Lottery_LimitedCharacter` | Shared |
| Gashapon | Arc Miracle Box | `Arc_MiracleBox` | Shared |
| Gashapon | Mystery Box | `Gashapon_MysteryBox` | Per-rotation |
## What It Does
The exporter decodes Permanent Board, Limited Character Board, and Arc Miracle Box history pages from captured UDP data, applies conservative timestamp-boundary handling, and writes sanitized JSON suitable for tracker import.
The exporter decodes Permanent Board, Limited Character Board, Arc Miracle Box, and Mystery Box history pages from captured UDP data, applies conservative timestamp-boundary handling, and writes sanitized JSON suitable for tracker import.
> [!NOTE]
> The import JSON contains decoded history rows and the shareable NTE user UID when it can be detected. It does **not** export tokens, account IDs, role IDs, device IDs, server IPs, raw packets, cookies, session data, or other capture metadata.
@ -96,6 +97,7 @@ Once running, open any supported history board in game. The tool keeps listening
- `<user_uid>_Permanent_<date_time>.json`
- `<user_uid>_Limited_<date_time>.json`
- `<user_uid>_Arc_<date_time>.json`
- `<user_uid>_MysteryBox_<date_time>.json`
If the user UID is not detected automatically, the console asks for it before saving. Leaving it blank saves as `unknown_<banner>_<date_time>.json`, but may prevent import on some trackers.
@ -199,7 +201,7 @@ History always loads page 1 first and is scrolled downward, so the exporter anch
Within a timestamp group, ordinal 0 is the newest record and unseen rows can only append after the captured ones, so **every exported UID is stable** — including a partially captured oldest 10-pull. All decoded rows are therefore exported. Re-scanning later simply adds any rows that were not yet captured, with the same UIDs for the rows already seen.
For Monopoly, Points Gift and Chase Reward rows stay in the timestamp group for UID ordinal generation, but only `result_type = dice` rows count toward pull-set sizing. Arc pulls are always 10-pulls. In both systems every captured group is exported, including the oldest one even if it is a partially captured pull set, because its captured prefix is ordinal-stable.
For Monopoly, Points Gift and Chase Reward rows stay in the timestamp group for UID ordinal generation, but only `result_type = dice` rows count toward pull-set sizing. Arc pulls are always 10-pulls. Mystery Box records are single pulls and the final page may contain fewer than five records. Every captured group is exported, including an incomplete oldest pull set, because its captured prefix is ordinal-stable.
## Adapters

View file

@ -15,7 +15,7 @@ decoder-only offsets.
"capture_source": "npcap",
"exporter": {
"name": "nte-history-exporter",
"version": "0.1.10"
"version": "0.1.11"
},
"banner": {
"id": "Lottery_Permanent",
@ -71,6 +71,8 @@ Current stable pool IDs:
- `Lottery_LimitedCharacter`: Limited Character Board. New limited character
banners should still use this ID while they share the same history/pity pool.
- `Arc_MiracleBox`: Arc Miracle Box.
- `Gashapon_MysteryBox`: Mystery Box. Records are single pulls and history is
not split into event rotations by the exporter.
Avoid using `banner.name`, `reward_name`, `reward_type` or `reward_rank` as primary IDs. They
are useful display fields, but may change when mapping files are updated.
@ -110,6 +112,9 @@ Arc source:
nte|gashapon|pool_group_id|timestamp_raw|timestamp_group_ordinal
```
Mystery Box uses the same Gashapon UID source. Reward quantity is preserved,
but each record represents exactly one pull and has `result_type = single_pull`.
## Example records
Monopoly:
@ -146,6 +151,24 @@ Arc:
}
```
Mystery Box:
```json
{
"uid": "2610c6e96afd64aa4a53fb9e8cebe031",
"pool_group_id": "Gashapon_MysteryBox",
"timestamp": "2026-07-08 19:09:33",
"timestamp_group_ordinal": 0,
"result_type": "single_pull",
"reward_type": "item",
"reward_id": "vehicle039",
"reward_name": "Draco",
"reward_rank": "S",
"quantity": 1,
"source_type": "mystery_box"
}
```
## CSV diagnostics
CSV exports are mainly for debugging the decoder. Tools should prefer JSON

View file

@ -1,10 +1,11 @@
# Limitations
This is a prototype and supports `Lottery_Permanent`, `Lottery_LimitedCharacter`, and `Arc_MiracleBox`.
This is a prototype and supports `Lottery_Permanent`, `Lottery_LimitedCharacter`, `Arc_MiracleBox`, and `Gashapon_MysteryBox`.
Known limitations:
- Other NTE banners are not implemented yet.
- Mystery Box history does not expose a semantic rotation ID in each decoded row. The exporter deliberately keeps one unsplit history stream and leaves rotation grouping to consumers.
- The game appears not to provide a unique server-side roll ID in the decoded record body.
- UIDs are generated deterministically from decoded fields and timestamp-group order.
- A partially captured oldest timestamp group is still exported; its captured prefix has stable UIDs, and a later deeper scan adds the rest with the same UIDs.

View file

@ -76,6 +76,13 @@ from the committed reward mappings. It still writes the staged artifacts.
- Other pull rewards are selected by `GachaIllustrate` and resolved to the
inventory tables for quality and localization keys. Appearance-table IDs
provide canonical casing for glider rewards.
- Mystery Box rewards are selected from every row in
`DT_GashaponLotteryGlobal`. No event IDs are hard-coded, so future rotations
are included automatically. Rewards resolve through the inventory, capital,
appearance, and vehicle-item tables using case-insensitive IDs.
- Fashion, avatar-frame, and business-card Mystery Box rewards are categorized
as cosmetics; other Mystery Box rewards, including vehicle keys and
currencies, are categorized as items.
- Names are resolved strictly through `Localization/en/game.json`. A missing or
ambiguous key fails the update instead of falling back to a DT value.
- Character-awakening illustration entries are not independent pull rewards

View file

@ -1,6 +1,6 @@
# Packet Format Notes
This prototype supports separate Monopoly and Arc/Gashapon history decoders.
This prototype supports separate Monopoly, Arc/Gashapon, and Mystery Box history decoders.
## Decoder strategy
@ -105,3 +105,15 @@ Pages are anchored to the continuous run starting at page 1 (history always load
- Arc timestamps use `unix_seconds = little_endian_u64(timestamp_raw) / 20000000 - 62135596800`.
- Arc pulls are treated as 10-pull timestamp groups. Like Monopoly, every captured group is exported, including the oldest one even if the scan stopped mid-10-pull (its captured prefix is ordinal-stable).
- Arc rows use the same `reward_type`, `reward_id`, `reward_name`, `reward_rank`, and `reward_key_hex` fields as Monopoly rows.
## Mystery Box
- Mystery Box requests use a 54-byte prefix.
- Request constant: `2060` / `0x080c` at offset 26.
- Request kind: `2110` / `0x083e` at offset 35.
- Request cursor: offset 31, with a step of `2`; page is `cursor / 2`.
- Responses contain an `FGashaponLotteryRecordData` structured block.
- Each row contains the reward ID, exact quantity, a record flag, and a standard .NET timestamp.
- Every row is one single pull regardless of its reward quantity.
- A full response page contains 5 rows; the final page may contain 1–4 rows.
- History is exported under `Gashapon_MysteryBox` without attempting to infer or split rotations.

View file

@ -21,5 +21,20 @@
"Fashion_vehicle_1071_V010": {"type": "cosmetic", "name": "Hound Blitz - Livery", "rank": "S"},
"Fashion_vehicle_1075_V006": {"type": "cosmetic", "name": "Hidden Dragon - Livery", "rank": "S"},
"Fashion_vehicle_1076_V024": {"type": "cosmetic", "name": "Hidden Dragon - Livery", "rank": "S"},
"Fashion_vehicleSkin_1052_V016": {"type": "cosmetic", "name": "Golden Breeze - Livery", "rank": "S"}
"Fashion_vehicleSkin_1052_V016": {"type": "cosmetic", "name": "Golden Breeze - Livery", "rank": "S"},
"vehicle039": {"type": "item", "name": "Draco", "rank": "S"},
"Fashion_1051_rpg_level3_Dye1": {"type": "cosmetic", "name": "Glory Days - Braveheart", "rank": "S"},
"Fashion_1051_rpg_level3_Dye2": {"type": "cosmetic", "name": "Glory Days - Starfall", "rank": "S"},
"Fashion_1051_rpg_level3_Base": {"type": "cosmetic", "name": "Glory Days", "rank": "S"},
"SpecialGift_ticket": {"type": "item", "name": "Floe Cinema Ticket", "rank": "A"},
"Fons": {"type": "item", "name": "Fons", "rank": "A"},
"CharacterUpMaterial_lv3": {"type": "item", "name": "Elite Hunter Guide", "rank": "A"},
"Gold": {"type": "item", "name": "Beetle Coin", "rank": "B"},
"CityAbility_UpMaterial": {"type": "item", "name": "Dreamless Seed", "rank": "B"},
"SpecialGift_letter": {"type": "item", "name": "A Handwritten Letter", "rank": "S"},
"Frame_MangHe_shinku": {"type": "cosmetic", "name": "Ember Trail", "rank": "S"},
"Fashion_character_1051_07": {"type": "cosmetic", "name": "Perfect Partners", "rank": "S"},
"vehicle026": {"type": "item", "name": "Porsche 918 Spyder", "rank": "S"},
"Frame_MangHe_bsj": {"type": "cosmetic", "name": "Heartbeat Moment", "rank": "S"},
"BussinessCard_MangHe_bsj": {"type": "cosmetic", "name": "Trackside Shadow", "rank": "S"}
}

34
mappings/mystery_box.json Normal file
View file

@ -0,0 +1,34 @@
{
"pool_key": "mystery_box",
"game": "Neverness to Everness",
"system": {
"id": "gashapon",
"name": "Gashapon"
},
"banner": {
"id": "Gashapon_MysteryBox",
"name": "Mystery Box",
"shared_pity": false
},
"request": {
"family": "mystery_box",
"length": 54,
"constant": 2060,
"constant_offset": 26,
"request_kind": 2110,
"request_kind_offset": 35,
"cursor_offset": 31,
"cursor_step": 2,
"page_formula": "cursor / 2"
},
"response": {
"records_per_full_page": 5,
"marker": "FGashaponLotteryRecordData",
"pulls_per_record": 1,
"final_page_may_be_partial": true
},
"notes": [
"Mystery Box is a finite 15-draw prize pool whose rewards are removed as they are drawn.",
"History is exported as one pull per record and is not split into rotations by the exporter."
]
}

View file

@ -1,6 +1,6 @@
[project]
name = "nte-history-exporter"
version = "0.1.10"
version = "0.1.11"
description = "Cross-platform Neverness to Everness pull-history exporter."
requires-python = ">=3.10"
dependencies = []

View file

@ -1 +1 @@
__version__ = "0.1.10"
__version__ = "0.1.11"

View file

@ -7,6 +7,10 @@ from typing import Any
from nte_history_exporter.decoder.boundary import select_continuous_run_from_page_1
from nte_history_exporter.decoder.arc import build_arc_rows_from_pairs, select_continuous_arc_run
from nte_history_exporter.decoder.run import build_rows_from_pairs
from nte_history_exporter.decoder.mystery_box import (
build_mystery_box_rows_from_pairs,
select_continuous_mystery_box_run,
)
from nte_history_exporter.decoder.user_uid import extract_user_uid_candidates
from nte_history_exporter.live_capture.session import LiveHistorySession, UdpPacket
@ -101,12 +105,21 @@ def decode_mitmproxy_flows(path: str | Path, flow_index: int | None = None) -> d
packet = UdpPacket(ts, remote_ip, local_ip, remote_port, local_port, content)
session.process_packet(packet)
pairs = [pair for pair in session.pairs if pair[7] != "arc_miracle_box"]
pairs = [
pair
for pair in session.pairs
if pair[7] not in {"arc_miracle_box", "mystery_box"}
]
arc_pairs = [pair for pair in session.pairs if pair[7] == "arc_miracle_box"]
mystery_box_pairs = [pair for pair in session.pairs if pair[7] == "mystery_box"]
best_run, run_warnings = select_continuous_run_from_page_1(pairs)
rows_out = build_rows_from_pairs(best_run)
best_arc_run, arc_warnings = select_continuous_arc_run(arc_pairs)
arc_rows = build_arc_rows_from_pairs(best_arc_run)
best_mystery_box_run, mystery_box_warnings = select_continuous_mystery_box_run(
mystery_box_pairs
)
mystery_box_rows = build_mystery_box_rows_from_pairs(best_mystery_box_run)
return {
"flow_index": resolved_flow_index,
@ -118,6 +131,10 @@ def decode_mitmproxy_flows(path: str | Path, flow_index: int | None = None) -> d
"best_arc_run": best_arc_run,
"arc_rows": arc_rows,
"arc_warnings": arc_warnings,
"mystery_box_pairs": mystery_box_pairs,
"best_mystery_box_run": best_mystery_box_run,
"mystery_box_rows": mystery_box_rows,
"mystery_box_warnings": mystery_box_warnings,
"user_uid": session.user_uid or user_uid,
"capture_diagnostics": session.diagnostic_report(),
}

View file

@ -66,7 +66,13 @@ def main(argv: list[str] | None = None) -> int:
return 1
decoded = decode_mitmproxy_flows(args.capture_source, args.flow_index)
if decoded["arc_rows"] and not decoded["rows"]:
if decoded["mystery_box_rows"] and not decoded["rows"] and not decoded["arc_rows"]:
rows = decoded["mystery_box_rows"]
warnings = decoded["mystery_box_warnings"]
kind = "mystery_box"
best_run = decoded["best_mystery_box_run"]
pair_count = len(decoded["mystery_box_pairs"])
elif decoded["arc_rows"] and not decoded["rows"]:
rows = decoded["arc_rows"]
warnings = decoded["arc_warnings"]
kind = "arc_miracle_box"

View file

@ -82,6 +82,7 @@ def print_live_instructions(local_ip: str, backend: str = "windows_raw", detail:
print(style(" Monopoly > Standard Board history", CYAN))
print(style(" Monopoly > Limited Character Board history", CYAN))
print(style(" Gashapon > Arc Miracle Box history", CYAN))
print(style(" Gashapon > Mystery Box history", CYAN))
print(" 3. Start at page 1 and scroll down through every page")
print(" you want exported.")
print(" 4. Scroll one page past where you plan to stop so the")

View file

@ -10,8 +10,10 @@ LIMITED_CHARACTER_BANNER_ID = "Lottery_LimitedCharacter"
LIMITED_CHARACTER_BANNER_NAME = "Limited Character Board"
ARC_BANNER_ID = "Arc_MiracleBox"
ARC_BANNER_NAME = "Arc Miracle Box"
MYSTERY_BOX_BANNER_ID = "Gashapon_MysteryBox"
MYSTERY_BOX_BANNER_NAME = "Mystery Box"
EXPORTER_NAME = "nte-history-exporter"
EXPORTER_VERSION = "0.1.10"
EXPORTER_VERSION = "0.1.11"
HISTORY_REQUEST_BANNER = 4220
HISTORY_REQUEST_LENGTH = 45
@ -38,6 +40,12 @@ POOL_META = {
"system": ARC_SYSTEM_NAME,
"shared_pity": True,
},
"mystery_box": {
"id": MYSTERY_BOX_BANNER_ID,
"name": MYSTERY_BOX_BANNER_NAME,
"system": ARC_SYSTEM_NAME,
"shared_pity": False,
},
}
ARC_HISTORY_REQUEST_BANNER = 2060
@ -47,6 +55,15 @@ ARC_HISTORY_PAGE_CURSOR_MULTIPLIER = 2
ARC_RESPONSE_FIRST_RECORD_OFFSET = 0x4C
ARC_TIMESTAMP_TICKS_PER_SECOND = 20_000_000
MYSTERY_BOX_HISTORY_REQUEST_LENGTH = 54
MYSTERY_BOX_HISTORY_REQUEST_CONSTANT_OFFSET = 26
MYSTERY_BOX_HISTORY_REQUEST_CONSTANT = 2060
MYSTERY_BOX_HISTORY_CURSOR_OFFSET = 31
MYSTERY_BOX_HISTORY_PAGE_CURSOR_MULTIPLIER = 2
MYSTERY_BOX_HISTORY_REQUEST_KIND_OFFSET = 35
MYSTERY_BOX_HISTORY_REQUEST_KIND = 2110
MYSTERY_BOX_MARKER = b"FGashaponLotteryRecordData"
TIMESTAMP_TICKS_PER_SECOND = 40_000_000
DOTNET_UNIX_EPOCH_SECONDS = 62_135_596_800

View file

@ -0,0 +1,208 @@
from __future__ import annotations
import hashlib
import struct
from collections import defaultdict
from datetime import datetime, timezone
from typing import Any, Iterator
from nte_history_exporter.constants import (
ARC_SYSTEM,
DOTNET_UNIX_EPOCH_SECONDS,
GAME_UID_PART,
MYSTERY_BOX_BANNER_ID,
MYSTERY_BOX_HISTORY_CURSOR_OFFSET,
MYSTERY_BOX_HISTORY_PAGE_CURSOR_MULTIPLIER,
MYSTERY_BOX_HISTORY_REQUEST_CONSTANT,
MYSTERY_BOX_HISTORY_REQUEST_CONSTANT_OFFSET,
MYSTERY_BOX_HISTORY_REQUEST_KIND,
MYSTERY_BOX_HISTORY_REQUEST_KIND_OFFSET,
MYSTERY_BOX_HISTORY_REQUEST_LENGTH,
MYSTERY_BOX_MARKER,
POOL_META,
)
from nte_history_exporter.decoder.boundary import select_continuous_run_from_page_1
from nte_history_exporter.decoder.protocol import infer_reward_type
from nte_history_exporter.decoder.run import fmt_packet_time
from nte_history_exporter.mappings import REWARDS_BY_ID
DOTNET_TICKS_PER_SECOND = 10_000_000
MAX_RECORDS_PER_BLOCK = 100
MAX_REWARD_ID_LENGTH = 256
REWARDS_BY_CASEFOLDED_ID = {
reward_id.casefold(): reward for reward_id, reward in REWARDS_BY_ID.items()
}
def is_mystery_box_history_request(content: bytes) -> bool:
if len(content) < MYSTERY_BOX_HISTORY_REQUEST_LENGTH:
return False
return (
struct.unpack_from("<I", content, MYSTERY_BOX_HISTORY_REQUEST_CONSTANT_OFFSET)[0]
== MYSTERY_BOX_HISTORY_REQUEST_CONSTANT
and struct.unpack_from("<I", content, MYSTERY_BOX_HISTORY_REQUEST_KIND_OFFSET)[0]
== MYSTERY_BOX_HISTORY_REQUEST_KIND
)
def mystery_box_request_page(content: bytes) -> int:
cursor = struct.unpack_from("<I", content, MYSTERY_BOX_HISTORY_CURSOR_OFFSET)[0]
return cursor // MYSTERY_BOX_HISTORY_PAGE_CURSOR_MULTIPLIER
def _protocol_views(payload: bytes) -> Iterator[bytes]:
yield payload
# History payloads can begin at a non-byte-aligned transport position.
for bit_shift in range(1, 8):
shifted = bytearray()
for index in range(max(0, len(payload) - 8)):
bit_pos = (8 + index) * 8 + bit_shift
byte_pos, shift = divmod(bit_pos, 8)
if byte_pos >= len(payload):
break
value = payload[byte_pos] >> shift
if shift and byte_pos + 1 < len(payload):
value |= payload[byte_pos + 1] << (8 - shift)
shifted.append(value & 0xFF)
yield bytes(shifted)
def _decode_timestamp(raw: bytes) -> tuple[int, float, str]:
ticks = struct.unpack("<Q", raw)[0]
unix_seconds = ticks / DOTNET_TICKS_PER_SECOND - DOTNET_UNIX_EPOCH_SECONDS
decoded = datetime.fromtimestamp(unix_seconds, timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
return ticks, unix_seconds, decoded
def _parse_view(data: bytes) -> list[dict[str, Any]]:
marker_pos = data.find(MYSTERY_BOX_MARKER)
if marker_pos < 0:
return []
pos = marker_pos + len(MYSTERY_BOX_MARKER)
if pos < len(data) and data[pos] == 0:
pos += 1
if pos + 12 > len(data):
return []
_reserved, _declared_size, row_count = struct.unpack_from("<III", data, pos)
pos += 12
if row_count > MAX_RECORDS_PER_BLOCK:
return []
rows: list[dict[str, Any]] = []
try:
for _ in range(row_count):
record_start = pos
reward_length = struct.unpack_from("<I", data, pos)[0]
pos += 4
if not 1 <= reward_length <= MAX_REWARD_ID_LENGTH or pos + reward_length + 13 > len(data):
return []
reward_raw = data[pos : pos + reward_length]
pos += reward_length
if not reward_raw.endswith(b"\0"):
return []
reward_id = reward_raw[:-1].decode("utf-8")
quantity = struct.unpack_from("<I", data, pos)[0]
pos += 4
record_flag = data[pos]
pos += 1
timestamp_raw = data[pos : pos + 8]
pos += 8
ticks, unix_seconds, timestamp_decoded = _decode_timestamp(timestamp_raw)
reward = REWARDS_BY_ID.get(reward_id) or REWARDS_BY_CASEFOLDED_ID.get(
reward_id.casefold(), {}
)
rows.append(
{
"record_start": record_start,
"record_end": pos,
"record_len": pos - record_start,
"reward_type": reward.get("type") or infer_reward_type(reward_id),
"reward_id": reward_id,
"reward_name": reward.get("name", ""),
"reward_rank": reward.get("rank"),
"quantity": quantity,
"source_type": "mystery_box",
"result_type": "single_pull",
"timestamp_raw_hex": timestamp_raw.hex(),
"timestamp_ticks": ticks,
"timestamp_unix": unix_seconds,
"timestamp_decoded": timestamp_decoded,
"record_flag": record_flag,
"record_hex": data[record_start:pos].hex(),
"decoder_mode": "structured",
}
)
except (OSError, OverflowError, UnicodeDecodeError, ValueError, struct.error):
return []
return rows
def parse_mystery_box_response(payload: bytes) -> list[dict[str, Any]]:
for view in _protocol_views(payload):
rows = _parse_view(view)
if rows:
return rows
return []
def make_mystery_box_uid(timestamp_raw: str, ordinal: int) -> str:
source = "|".join(
[GAME_UID_PART, ARC_SYSTEM, MYSTERY_BOX_BANNER_ID, timestamp_raw, str(ordinal)]
)
return hashlib.sha256(source.encode("utf-8")).hexdigest()[:32]
def annotate_mystery_box_rows(rows: list[dict[str, Any]]) -> None:
groups: dict[str, list[int]] = defaultdict(list)
for index, row in enumerate(rows):
groups[row["timestamp_raw_hex"]].append(index)
for group_index, (timestamp_raw, indexes) in enumerate(groups.items()):
for ordinal, index in enumerate(indexes):
row = rows[index]
row["timestamp_group_index"] = group_index
row["timestamp_group_ordinal"] = ordinal
row["timestamp_group_size_seen"] = len(indexes)
row["timestamp_group_record_size_seen"] = len(indexes)
row["timestamp_group_boundary"] = (
"oldest" if group_index == len(groups) - 1 else ("newest" if group_index == 0 else "")
)
row["uid"] = make_mystery_box_uid(timestamp_raw, ordinal)
row["uid_status"] = "stable"
row["export_record"] = True
row["skip_reason"] = ""
def build_mystery_box_rows_from_pairs(pairs: list[tuple]) -> list[dict[str, Any]]:
pool = POOL_META["mystery_box"]
rows: list[dict[str, Any]] = []
for pair in pairs:
page, offset, req_i, req_ts, resp_i, resp_ts, response = pair[:7]
records = parse_mystery_box_response(response)
if len(pair) > 9:
slice_start, slice_count = pair[8:10]
records = records[slice_start : slice_start + slice_count]
for row_index, record in enumerate(records, start=1):
rows.append(
{
**record,
"page": page,
"offset": offset,
"row": row_index,
"pool_group_id": pool["id"],
"pool_group_name": pool["name"],
"request_msg": req_i,
"request_time_utc": fmt_packet_time(req_ts),
"response_msg": resp_i,
"response_time_utc": fmt_packet_time(resp_ts),
"response_len": len(response),
"record_count": len(records),
}
)
annotate_mystery_box_rows(rows)
return rows
def select_continuous_mystery_box_run(
pairs: list[tuple],
) -> tuple[list[tuple], list[dict[str, Any]]]:
return select_continuous_run_from_page_1(pairs)

View file

@ -58,6 +58,7 @@ FIELDNAMES = [
"record_len",
"dice_offset_in_record",
"record_hex",
"record_flag",
]

View file

@ -9,6 +9,7 @@ from nte_history_exporter.constants import (
EXPORTER_NAME,
GAME_NAME,
POOL_META,
MYSTERY_BOX_BANNER_ID,
)
@ -83,6 +84,21 @@ def _record_for_export(row: dict[str, Any]) -> dict[str, Any]:
"source_type": row.get("source_type"),
}
if row.get("pool_group_id") == MYSTERY_BOX_BANNER_ID:
return {
"uid": row.get("uid"),
"pool_group_id": row.get("pool_group_id"),
"timestamp": row.get("timestamp_decoded"),
"timestamp_group_ordinal": row.get("timestamp_group_ordinal"),
"result_type": "single_pull",
"reward_type": row.get("reward_type"),
"reward_id": row.get("reward_id"),
"reward_name": row.get("reward_name"),
"reward_rank": row.get("reward_rank"),
"quantity": row.get("quantity"),
"source_type": row.get("source_type"),
}
return {
"uid": row.get("uid"),
"pool_group_id": row.get("pool_group_id", BANNER_ID),

View file

@ -23,6 +23,7 @@ EXPORT_PREFIXES = {
"permanent": "Permanent",
"limited_character": "Limited",
"arc_miracle_box": "Arc",
"mystery_box": "MysteryBox",
}
CAPTURE_SOURCE_LABELS = {
@ -148,7 +149,7 @@ def run_live_capture(
pairs = session.pairs_for_kind(kind)
best_run, run_warnings = select_continuous_run_from_page_1(pairs)
rows = session.build_rows(kind)
if kind != "arc_miracle_box":
if kind not in {"arc_miracle_box", "mystery_box"}:
rows = annotate_groups(rows)
warnings = run_warnings
pages_seen = [p[0] for p in best_run]

View file

@ -20,7 +20,15 @@ from nte_history_exporter.decoder.protocol import (
response_contains_history_marker,
)
from nte_history_exporter.decoder.run import build_rows_from_pairs
from nte_history_exporter.decoder.mystery_box import (
build_mystery_box_rows_from_pairs,
is_mystery_box_history_request,
mystery_box_request_page,
parse_mystery_box_response,
select_continuous_mystery_box_run,
)
from nte_history_exporter.decoder.structured_protocol import FORK_MARKER, MONOPOLY_MARKER
from nte_history_exporter.constants import MYSTERY_BOX_MARKER
from nte_history_exporter.decoder.user_uid import extract_user_uid_candidates
from nte_history_exporter.live_capture.diagnostics import CaptureDiagnostics
@ -165,6 +173,30 @@ class LiveHistorySession:
self.last_page_seen = req.page
return False
if packet.src_ip == self.local_ip and is_mystery_box_history_request(packet.payload):
page = mystery_box_request_page(packet.payload)
req = PendingRequest(
page=page,
offset=page * 2,
kind="mystery_box",
request_msg=self.packet_count,
request_time=packet.timestamp,
src_ip=packet.src_ip,
dst_ip=packet.dst_ip,
src_port=packet.src_port,
dst_port=packet.dst_port,
)
self._queue_request(req)
self.diagnostics.counters["history_requests_recognized"] += 1
self.diagnostics.add_event(
"HISTORY_REQUEST_RECOGNIZED",
self.packet_count,
kind=req.kind,
page=req.page,
)
self.last_page_seen = req.page
return False
if packet.dst_ip != self.local_ip:
return False
@ -191,12 +223,24 @@ class LiveHistorySession:
monopoly_records = decode_response_records(packet.payload)
arc_records = parse_arc_response(packet.payload) if not monopoly_records else []
mystery_box_records = (
parse_mystery_box_response(packet.payload)
if not monopoly_records and not arc_records
else []
)
if monopoly_records:
candidates = [req for req in connection_candidates if req.kind != "arc_miracle_box"]
candidates = [
req
for req in connection_candidates
if req.kind not in {"arc_miracle_box", "mystery_box"}
]
records = monopoly_records
elif arc_records:
candidates = [req for req in connection_candidates if req.kind == "arc_miracle_box"]
records = arc_records
elif mystery_box_records:
candidates = [req for req in connection_candidates if req.kind == "mystery_box"]
records = mystery_box_records
else:
candidates = connection_candidates
records = []
@ -215,6 +259,7 @@ class LiveHistorySession:
if response_contains_history_marker(packet.payload)
or MONOPOLY_MARKER in packet.payload
or FORK_MARKER in packet.payload
or MYSTERY_BOX_MARKER in packet.payload
else "NO_HISTORY_MARKER"
)
self._record_rejected_candidate(candidates, packet.payload, reason)
@ -340,6 +385,9 @@ class LiveHistorySession:
if kind == "arc_miracle_box":
best_run, _warnings = select_continuous_arc_run(self.pairs_for_kind(kind))
return build_arc_rows_from_pairs(best_run)
if kind == "mystery_box":
best_run, _warnings = select_continuous_mystery_box_run(self.pairs_for_kind(kind))
return build_mystery_box_rows_from_pairs(best_run)
return build_rows_from_pairs(self.best_run(kind))
def best_run(self, kind: str | None = None) -> list[tuple]:

View file

@ -19,6 +19,8 @@ ASSET_TABLES = {
"capital_inventory": "DataTable/Inventory/DT_CapitalItemConfig.json",
"appearances": "DataTable/Character/Appearance/DT_AppearanceData.json",
"illustrations": "DataTable/Gacha/GachaIllustrate.json",
"mystery_box_pools": "DataTable/GashaponLottery/DT_GashaponLotteryGlobal.json",
"vehicle_inventory": "DataTable/Vehicle/DT_VehicleItemData.json",
"localization": "Localization/en/game.json",
}
REWARD_MAPPING_FILES = ("arcs.json", "characters.json", "items.json")
@ -282,27 +284,60 @@ def _build_item_mapping(
inventory = _casefold_index(tables["inventory"], "inventory")
capital = _casefold_index(tables["capital_inventory"], "capital inventory")
appearances = _casefold_index(tables["appearances"], "appearances")
vehicle_inventory = _casefold_index(tables["vehicle_inventory"], "vehicle inventory")
result: dict[str, dict[str, Any]] = {}
for illustrated_id in tables["illustrations"]:
folded = illustrated_id.casefold()
if illustrated_id.isdigit() or folded.startswith("fork_") or folded.startswith("characterawaken_"):
candidate_ids = _reward_candidate_ids(tables)
for candidate_id in candidate_ids:
folded = candidate_id.casefold()
if candidate_id.isdigit() or folded.startswith("fork_") or folded.startswith("characterawaken_"):
continue
sources = (inventory, capital, appearances)
sources = (inventory, capital, appearances, vehicle_inventory)
match = next((source.get(folded) for source in sources if folded in source), None)
if match is None:
raise MappingUpdateError(f"illustrated reward {illustrated_id} is missing from item tables")
raise MappingUpdateError(f"reward {candidate_id} is missing from item tables")
canonical_id, row = match
if folded.startswith("fashion_glide_") and folded in appearances:
canonical_id = appearances[folded][0]
meta = _normalise_row(canonical_id, row, translations)
result[canonical_id] = {
"type": "cosmetic" if canonical_id.casefold().startswith("fashion_") else "item",
"type": _item_mapping_type(canonical_id),
"name": meta["name"],
"rank": meta["rank"],
}
return dict(sorted(result.items(), key=lambda pair: (pair[0].casefold(), pair[0])))
def _reward_candidate_ids(tables: dict[str, dict[str, Any]]) -> list[str]:
"""Return rewards used by ordinary Gacha and every Mystery Box rotation."""
candidates: dict[str, str] = {
item_id.casefold(): item_id for item_id in tables["illustrations"]
}
for pool_id, pool in tables["mystery_box_pools"].items():
if not isinstance(pool, dict):
raise MappingUpdateError(f"Mystery Box pool {pool_id} must be an object")
gifts = pool.get("GiftList")
if not isinstance(gifts, list):
raise MappingUpdateError(f"Mystery Box pool {pool_id} has no GiftList")
for index, gift in enumerate(gifts):
if not isinstance(gift, dict):
raise MappingUpdateError(
f"Mystery Box pool {pool_id} gift {index} must be an object"
)
item_id = gift.get("ItemID")
if not isinstance(item_id, str) or not item_id:
raise MappingUpdateError(
f"Mystery Box pool {pool_id} gift {index} has no ItemID"
)
candidates.setdefault(item_id.casefold(), item_id)
return sorted(candidates.values(), key=lambda value: (value.casefold(), value))
def _item_mapping_type(item_id: str) -> str:
folded = item_id.casefold()
cosmetic_prefixes = ("fashion_", "frame_", "bussinesscard_")
return "cosmetic" if folded.startswith(cosmetic_prefixes) else "item"
def _casefold_index(rows: dict[str, Any], label: str) -> dict[str, tuple[str, Any]]:
result: dict[str, tuple[str, Any]] = {}
for item_id, row in rows.items():

View file

@ -8,6 +8,7 @@ POOL_MAPPING_FILES = {
"permanent": "permanent_board.json",
"limited_character": "limited_character_board.json",
"arc_miracle_box": "arc_miracle_box.json",
"mystery_box": "mystery_box.json",
}

View file

@ -0,0 +1 @@
[{"Rows":{}}]

View file

@ -0,0 +1 @@
[{"Rows":{}}]

View file

@ -72,6 +72,51 @@ class MappingUpdateTests(unittest.TestCase):
self.assertNotIn("UnusedItem", result.mappings["items.json"])
self.assertFalse(any(key.startswith("Characterawaken_") for key in result.mappings["items.json"]))
def test_future_mystery_box_pool_is_discovered_without_hard_coded_event_id(self):
assets = load_assets(assets_root=SAMPLE_ASSETS)
tables = deepcopy(assets.tables)
tables["mystery_box_pools"]["MangHe_wowzers"] = {
"GiftList": [
{"ItemID": "WowTicket"},
{"ItemID": "Frame_Wowzers"},
{"ItemID": "VehicleWow"},
]
}
tables["inventory"]["WowTicket"] = {
"ItemName": {"TableId": "/Game/Text/ST_Item.ST_Item", "Key": "wow_ticket"},
"ItemQuality": "EItemQuality::ITEM_QUALITY_PURPLE",
}
tables["inventory"]["Frame_Wowzers"] = {
"ItemName": {"TableId": "/Game/Text/ST_Item.ST_Item", "Key": "wow_frame"},
"ItemQuality": "EItemQuality::ITEM_QUALITY_BLUE",
}
tables["vehicle_inventory"]["vehiclewow"] = {
"ItemName": {
"TableId": "/Game/Text/ST_VehicleData.ST_VehicleData",
"Key": "vehicle_wow",
},
"ItemQuality": "EItemQuality::ITEM_QUALITY_ORANGE",
}
tables["localization"]["ST_Item"].update(
{"wow_ticket": "Wowzers Ticket", "wow_frame": "Wowzers Frame"}
)
tables["localization"]["ST_VehicleData"] = {"vehicle_wow": "Wowmobile"}
result = build_mapping_update(sample_current(), replace(assets, tables=tables))
self.assertEqual(
result.mappings["items.json"]["WowTicket"],
{"type": "item", "name": "Wowzers Ticket", "rank": "A"},
)
self.assertEqual(
result.mappings["items.json"]["Frame_Wowzers"],
{"type": "cosmetic", "name": "Wowzers Frame", "rank": "B"},
)
self.assertEqual(
result.mappings["items.json"]["vehiclewow"],
{"type": "item", "name": "Wowmobile", "rank": "S"},
)
def test_embedded_localized_strings_are_ignored(self):
_current, result = build_sample_update()

View file

@ -0,0 +1,116 @@
from tests.support import * # noqa: F401,F403
from datetime import datetime, timezone
from nte_history_exporter.constants import MYSTERY_BOX_MARKER
from nte_history_exporter.decoder.mystery_box import (
build_mystery_box_rows_from_pairs,
is_mystery_box_history_request,
mystery_box_request_page,
parse_mystery_box_response,
)
def mystery_box_request(page: int) -> bytes:
request = bytearray(54)
request[26:30] = (2060).to_bytes(4, "little")
request[31:35] = (page * 2).to_bytes(4, "little")
request[35:39] = (2110).to_bytes(4, "little")
request[40:44] = (2).to_bytes(4, "little")
request[44:48] = (2060).to_bytes(4, "little")
request[49:53] = (10).to_bytes(4, "little")
request[53] = 6
return bytes(request)
def mystery_box_response(records: list[tuple[str, int, datetime]]) -> bytes:
body = bytearray()
for reward_id, quantity, timestamp in records:
encoded_id = reward_id.encode("utf-8") + b"\0"
ticks = int(timestamp.timestamp() * 10_000_000) + 621_355_968_000_000_000
body += len(encoded_id).to_bytes(4, "little")
body += encoded_id
body += quantity.to_bytes(4, "little")
body += b"\x01"
body += ticks.to_bytes(8, "little")
return (
bytes(50)
+ MYSTERY_BOX_MARKER
+ b"\0"
+ bytes(4)
+ len(body).to_bytes(4, "little")
+ len(records).to_bytes(4, "little")
+ body
+ b"\x03"
)
class MysteryBoxDecodingTests(unittest.TestCase):
def test_recognizes_request_and_page_cursor(self):
request = mystery_box_request(6)
self.assertTrue(is_mystery_box_history_request(request))
self.assertEqual(mystery_box_request_page(request), 6)
def test_decodes_exact_quantity_and_timestamp(self):
timestamp = datetime(2026, 7, 8, 19, 9, 33, tzinfo=timezone.utc)
rows = parse_mystery_box_response(
mystery_box_response(
[
("vehicle039", 1, timestamp),
("SpecialGift_ticket", 3, timestamp),
("gold", 100_000, timestamp),
]
)
)
self.assertEqual(len(rows), 3)
self.assertEqual(rows[0]["reward_id"], "vehicle039")
self.assertEqual(rows[0]["reward_name"], "Draco")
self.assertEqual(rows[0]["timestamp_decoded"], "2026-07-08 19:09:33")
self.assertEqual(rows[1]["quantity"], 3)
self.assertEqual(rows[1]["result_type"], "single_pull")
self.assertEqual(rows[2]["reward_name"], "Beetle Coin")
self.assertEqual(rows[2]["reward_rank"], "B")
def test_live_session_accepts_partial_final_page(self):
session = LiveHistorySession("192.168.0.10")
request = mystery_box_request(3)
timestamp = datetime(2026, 7, 8, 19, 9, 33, tzinfo=timezone.utc)
response = mystery_box_response(
[("Fons", 100_000, timestamp), ("gold", 100_000, timestamp)]
)
self.assertFalse(
session.process_packet(
UdpPacket(1.0, "192.168.0.10", "203.0.113.5", 50000, 40000, request)
)
)
self.assertTrue(
session.process_packet(
UdpPacket(1.1, "203.0.113.5", "192.168.0.10", 40000, 50000, response)
)
)
self.assertEqual(session.kinds_seen(), ["mystery_box"])
self.assertEqual(session.pairs[0][0], 3)
self.assertEqual(session.pairs[0][8:10], (0, 2))
def test_rows_and_export_use_non_shared_mystery_box_banner(self):
timestamp = datetime(2026, 7, 8, 19, 9, 33, tzinfo=timezone.utc)
response = mystery_box_response([("Fons", 100_000, timestamp)])
rows = build_mystery_box_rows_from_pairs(
[(1, 2, 1, 1.0, 2, 1.1, response, "mystery_box", 0, 1)]
)
export = build_export_json(rows, [])
self.assertEqual(export["banner"]["id"], "Gashapon_MysteryBox")
self.assertEqual(export["banner"]["name"], "Mystery Box")
self.assertIs(export["banner"]["shared_pity"], False)
self.assertEqual(export["records"][0]["result_type"], "single_pull")
self.assertEqual(export["records"][0]["quantity"], 100_000)
self.assertNotIn("roll_result", export["records"][0])
self.assertEqual(rows[0]["timestamp_group_size_seen"], 1)
self.assertEqual(rows[0]["uid_status"], "stable")
def test_export_path_uses_mystery_box_prefix(self):
_csv_path, json_path = export_paths("mystery_box", "218216016349")
self.assertRegex(
json_path.name,
r"^218216016349_MysteryBox_\d{8}_\d{6}(?:_\d+)?\.json$",
)