metakit
metakit is a tool used to create a unified definition for game metadata in all systems supported by retrokit. Since metadata is pulled from a variety of sources, it’s important to define a standard schema so that all downstream tools (like romkit) can use that metadata without having to worry about what system it’s dealing with.
Design
metakit is designed with a few goals in mind:
- Metadata for a game and all of its clones are grouped together
- As much metadata as possible is pulled from external sources (rather than managed manually)
- Metadata files are validated, maintained, and updated in an automated fashion
- All changes to metadata groups are reflected within retrokit
These goals are intended to help ensure that this data is high-quality and can be kept up-to-date as system DAT files are updated.
Dependencies
The minimum metakit dependencies are:
- internetarchive
- questionary
- waybackpy
To install the relevant Python dependencies:
# Install the minimum Python dependencies
metakit/setup.sh depends
Usage
# Downloads any external metadata that hasn't already been downloaded (this will not update)
bin/metakit.sh cache_external_data <system>
# Reformats the system metadata file, keeping the data clean
bin/metakit.sh format <system>
# Removes unneeded data from the metadata and scraper files (e.g. data for games that no longer exist)
bin/metakit.sh vacuum <system>
# Forces any external data used by the system to be re-downloaded
bin/metakit.sh recache_external_data <system>
# Forces all scraped metadata to be refreshed from the source
bin/metakit.sh rescrape <system>
# Forces machines that have incomplete scrape metadata to be refreshed from the source
bin/metakit.sh scrape_incomplete <system>
# Forces machines that have were never found in the scraper sources
bin/metakit.sh scrape_missing <system>
# Brings all romset dats up-to-date
bin/metakit.sh update_dats <system>
# Migrates groups to their current names as defined by the system's DAT
# files and the priority settings defined in the metakit configuraiton
bin/metakit.sh update_groups <system>
# Discovers new manuals on preconfigured website sources that have been added
# since the given date.
#
# The date passed in should be the date of existing snapshots (if available).
bin/metakit.sh find_manuals <system> [<date>] [<website_name>]
# Snapshots all manual website sources so that they can be compared against
# on a future date to see if any new manuals have been added
#
# The date passed in should be the *current* date (i.e. when the content was
# downloaded)
bin/metakit.sh snapshot_manuals <system> <date>
# Update the content of the database based on internal/external data
bin/metakit.sh update_metadata <system>
# Runs the following (in order):
# * update_dats
# * update_groups
# * vacuum
# * recache_external_data
# * scrape
# * update_metadata
bin/metakit.sh update <system>
# Validate the content of the system's database
bin/metakit.sh validate <system>
# Validates that machines are mapped properly to names in each romset's discovery data.
# That is, this ensures that the machine is found in any external archives configured
# for a system (based on either the machine's name or its alternates).
bin/metakit.sh validate_discovery <system>
Supported metadata
The following types of metadata are currently supported.
Age Rating
The maturity rating for the game.
Property: age_rating
Systems:
- all
Sources:
Aliases
Other titles that are associated with the game via the parent->child datfile relationship.
Property: aliases
Systems:
- all
Sources:
- Romsets
Alternate Names
Other names the game may be referred to as. This can be useful if an archive source is different from the latest DAT file for a system.
Property: alternates
Systems:
- all
Sources:
- User-defined
Buttons
The action associated with each button on the controller.
Property: buttons
Systems:
- arcade
Sources:
- https://github.com/Texacate/Visual-RetroPie-Control-Maps
Category
The type of rom (e.g. game, tool, etc.).
Property: category
Systems:
- All cd-based systems (via redump)
Sources:
- Redump DAT files
Controls
The types of controllers used (e.g. lightgun, joy, etc.).
Property: controls
Systems:
- arcade
- lightgun-supported systems
Sources:
- MAME
- Sinden Lightgun compatiblity spreadsheet
Custom
An open-ended dictionary that allows you to define key-value attributes on a group
rather than open-ended strings like you can with tags.
Property: custom
Systems:
- all
Sources:
- User-provided
Developer
The name of the company that developed the game.
Property: developer
Systems:
- all
Sources:
- MAME
- Screenscraper
Discs
The number of discs associated with the title. Note that this is based on the primary region (in case you see discrepancies between regions).
Property: discs
Systems:
- All cd-based systems
Sources:
- Discs
Emulation
The preferred emulator and compatibility rating.
Property: emulation
Systems:
- arcade
- atari5200
- atarijaguar
- dreamcast
- n64
- nds
- psp
- psx
- saturn
Sources:
Genres
The genres associated with the game.
Property: genres
Systems:
- all
Sources:
Languages
The languages used within the game.
Property: languages
Systems:
- arcade
Sources:
Manuals
Manuals available for the game.
Property: manuals
Systems:
- all
Sources:
- Various
Mechanical
Whether the game requires mechanical features.
Property: mechanical
Systems:
- arcade
- gb
Sources:
- MAME
Media
Additional media used for the game, such as: artwork, overlay, etc.
Property: media
Systems:
- all
Sources:
Merge
Additional game titles to mark as clones when not specified explicitly in the system’s DAT files.
Property: merge
Systems:
- all
Sources:
There are important differences between what’s considered the parent and what’s considered the clone between different systems.
No-Intro DAT files generally sort games based on the rules laid out here:
- Final/Complete > Proto/Beta/Demo
- Games containing En language > Other languages
- World > Continent/Multi Country > Country
- Old “main” console regions (EUR/USA/JPN) > Other countries (so i.e. Japan > Spain)
- Country with earlier dump available
- Highest revision
On the other hand, Redump DAT files are sorted chronologically based on the order in which they were dumped. Additionally, Redump does not provide clone metadata for its ROMs.
In order to (a) provide some consistency, (b) provide some stability in the metadata, and (c) make it easier to work with, the same general rules are applied to the merge groups generated from Redump DAT files.
Peripherals
System peripherals supported, such as link_cable, multitap, etc.
Property: peripherals
Systems:
- all
Sources:
Players
Maximum number of players supported.
Property: players
Systems:
- all
Sources:
Publisher
The company that published the game.
Property: publisher
Systems:
- all
Sources:
Rating
Community-based rating.
Property: rating
Systems:
- all
Sources:
Screen
Details about the screen (e.g. orientation).
Property: screen
Systems:
- arcade
- wonderswan
- wonderswancolor
Sources:
- MAME
Series
A series name to associate multiple games together.
Property: series
Systems:
- arcade
Sources:
Tags
Custom, user-defined tags.
Property: tags
Systems:
- all
Sources:
Year
Release year.
Property: year
Systems:
- all
Sources:
- MAME
- Screenscraper
System-specific usage
pc
metakit settings are used to manage the pc dosbox configuration files that are
mapped from the eXoDOS project. To generate new configurations:
rm -f config/systems/pc/configs/*.conf
PROFILES=metakit,filter-reset EXODOS_V6_HOME=... bin/romkit.sh install pc resources=config
PROFILES=metakit,filter-reset EXODOS_IGNORE_MISSING_CONFIGS=true EXODOS_V6_HOME=... bin/romkit.sh install pc resources=config