Metapack Build Commands

new: Creating New Packages

mp new

The mp new command creates source package directories with a proper name, a .gitignore file, and optionally, example data, entries and code. Typical usage, for creating a new package with most of the example options, is

mp new -o metatab.org -d tutorial -T "Quickstart Example Package"

The -C option will set a configuration file, which is a Metatab file that with terms that are copied into the metadata.csv file of the new package. Currently, it copies a limited number of terms, including:

• Terms in the Contacts section

• Root.Space

• Root.Time

• Root.Grain

• Root.Variant

• Root.Version

usage: mp new [-h] [-o ORIGIN] -d DATASET [-t TIME] [-s SPACE] [-g GRAIN]
              [-v VARIANT] [-r REVISION] [-T TITLE] [-L] [-E] [-J]
              [--template TEMPLATE] [-C CONFIG]

-h, --help

show this help message and exit

-o <origin>, --origin <origin>

Dataset origin, usually a second-level domain name. Required

-d <dataset>, --dataset <dataset>

Main dataset name. Required

-t <time>, --time <time>

Temporal extents, usually a year, ISO8601 time, or interval.

-s <space>, --space <space>

Space, geographic extent, such as a name of a state or a Census geoid

-g <grain>, --grain <grain>

Grain, the type of entity a row represents

-v <variant>, --variant <variant>

Variant, any distinguishing string

-r <revision>, --revision <revision>

Version, defaults to 1

-T <title>, --title <title>

Set the title

-L, --pylib

Configure a pylib directory for Python code extensions

-E, --example

Add examples of resources

-J, --jupyter

Create a Jupyter notebook source package

--template <template>

Metatab file template, defaults to ‘metatab’

-C <config>, --config <config>

Path to config file. Defaults to ~/.metapack-defaults.csv or value of METAPACK_DEFAULTS env var.Sets defaults for specia root terms and the Contacts section.

update: Update Package Information

mp update

usage: mp update [-h] [-c | -n | -s | -P | -D] [-A] [-X] [-C] [-F]
                 [metatabfile]

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’

-h, --help

show this help message and exit

-c, --categories

Update categories, creating a new categories.csv metadata file

-n, --name

Update the Name from the Datasetname, Origin and Version terms

-s, --schemas

Rebuild the schemas for files referenced in the resource section

-P, --schema-properties

Load schema properties from generators and upstream sources

-D, --descriptions

Import descriptions for package references

-A, --alt-name

Move AltNames to column name

-X, --clean-properties

Remove unused columns in the schema, like AltName

-C, --clean

Clean schema before processing

-F, --force

Force the operation

Cache dir: /Users/eric/Library/Application Support/metapack/

build: Create distribution Packages

mp build

Build source packages.

The mp build program runs all of the resources listed in a Metatab file and produces one or more Metapack packages with those resources localized. It will always try to produce a Filesystem package, and may optionally produce Excel, Zip and CSV packages.

Typical usage is to be run inside a source package directory with

$ mp build

To build all of the package types:

$ mp build -fezc

By default, packages are built with versioned names. The --nonversion-name option will create file packages with non-versioned name, and the --nonversioned-link option will produce a non-versioned soft link pointing to the versioned file.

usage: mp build [-h] [-p PROFILE] [-D PACKAGE_DIRECTORY] [-F] [-R] [-n | -N]
                [-e] [-z] [-f] [-c] [--clean-cache] [-C]
                [metatabfile]

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’.

-h, --help

show this help message and exit

-p <profile>, --profile <profile>

Name of a BOTO or AWS credentails profile

-D <package_directory>, --package-directory <package_directory>

Write Zip, Excel and CSV packages to an alternate directory

-F, --force

Force some operations, like updating the name and building packages

-R, --reuse-resources

When building Filesystem package, try to reuse resources built in prior build

-n, --nonversion-name

Write file packages with non-versioned names

-N, --nonversion-link

Create links with nonversioned names to file packages

-e, --excel

Create an excel archive from a metatab file

-z, --zip

Create a zip archive from a metatab file

-f, --filesystem

Create a filesystem archive from a metatab file

-c, --csv

Create a CSV archive from a metatab file

--clean-cache

Clean the download cache

-C, --clean

For some operations, like updating schemas, clear the section of existing terms first

edit: Alter package metadata

mp edit

usage: mp edit [-h] [-e] {add,change,delete} ...

-h, --help

show this help message and exit

-e, --echo

Print the file after editing

mp edit add

usage: mp edit add [-h] [-a ARG] term value [metatabfile]

term

A fully qualified term. Prefix with a third component to set the section ( ‘Resources.Root.Datafile’ )

value

The term value, for add and edit

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’

-h, --help

show this help message and exit

-a <arg>, --arg <arg>

Add an additional argument: ‘Name=value’

mp edit change

usage: mp edit change [-h] term value [metatabfile]

term

A fully qualified term. Prefix with a third component to set the section ( ‘Resources.Root.Datafile’ )

value

The term value, for add and edit

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’

-h, --help

show this help message and exit

mp edit delete

usage: mp edit delete [-h] term [metatabfile]

term

A fully qualified term. Prefix with a third component to set the section ( ‘Resources.Root.Datafile’ )

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’

-h, --help

show this help message and exit

stats: Calculate dataset statistics

mp stats

usage: mp stats [-h] [-n ROWS] [-s SAMPLE] [-H HEAD] [-d] [-D] [-V] [-T]
                [metatabfile]

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’

-h, --help

show this help message and exit

-n <rows>, --rows <rows>

Estimate of number of rows, for sampling

-s <sample>, --sample <sample>

Sample a subset of <SAMPLE> rows

-H <head>, --head <head>

Use only the first <HEAD> rows. Can substitute for –rows

-d, --descriptive

Calculate descriptive stats; min, max, mean, std, quartiles

-D, --distribution

Calculate distribution stats; histogram, skew, kurtosis

-V, --values

Display a set of unique values

-T, --types

Intuit types instead

url: Add urls to metadata

mp url

usage: mp url [-h] [-a ADD | -E ENUMERATE | -S SCRAPE] [metatabfile]

metatabfile

Path or URL to a metatab file. If not provided, defaults to ‘metadata.csv’

-h, --help

show this help message and exit

-a <add>, --add <add>

Add a file or url to the resources. With a directory add a data files in the directory. If given a URL to a web page, will add all links that point to CSV, Excel Files and data files in ZIP files. (Caution: it will download and cache all of these files. )

-E <enumerate>, --enumerate <enumerate>

Enumerate the resources referenced from a URL. Does not alter the Metatab file

-S <scrape>, --scrape <scrape>

Scrape data and documentation URLs from a web page

Cache dir: /Users/eric/Library/Application Support/metapack/

s3: Create AWS S3 packages

mp s3

usage: mp s3 [-h] [-p PROFILE] [-s S3] [-F] [-C] [metatabfile]

metatabfile

Path to a Metatab file

-h, --help

show this help message and exit

-p <profile>, --profile <profile>

Name of a BOTO or AWS credentials profile

-s <s3>, --s3 <s3>

URL to S3 where packages will be stored

-F, --force

Force write for all files

-C, --credentials

Show S3 Credentials and exit. Eval this string to setup credentials in other shells.