Publishing Rust Crates: a Justfile Workflow

Publishing crate versions to a private registry is something I do pretty frequently. I have a certain setup, tooling and workflow I have built-up over time that works pretty well.

Two tools are at the center of everything: cargo (duh) and just.

just ("just a command runner") works similarly to make, but without the weird syntax of Makefiles (i.e. no "phony" targets). It doesn't have build-system functionality (i.e. it doesn't perform logic on which things need to be compiled based on file metadata), but as a result it's much nicer to use for defining repeatable commands.

cargo is, of course, the Rust build tool, and the cargo publish subcommand is used to publish crates to a registry:

$ cargo publish --help
Upload a package to the registry

Usage: cargo publish [OPTIONS]

Options:
  -q, --quiet                   Do not print cargo log messages
      --index <INDEX>           Registry index URL to upload the package to
      --token <TOKEN>           Token to use when uploading
      --no-verify               Don't verify the contents by building them
  -v, --verbose...              Use verbose output (-vv very verbose/build.rs output)
      --allow-dirty             Allow dirty working directories to be packaged
      --color <WHEN>            Coloring: auto, always, never
      --target <TRIPLE>         Build for the target triple
      --frozen                  Require Cargo.lock and cache are up to date
      --target-dir <DIRECTORY>  Directory for all generated artifacts
      --locked                  Require Cargo.lock is up to date
  -p, --package [<SPEC>]        Package to publish
      --manifest-path <PATH>    Path to Cargo.toml
      --offline                 Run without accessing the network
      --config <KEY=VALUE>      Override a configuration value
  -F, --features <FEATURES>     Space or comma separated list of features to activate
      --all-features            Activate all available features
  -Z <FLAG>                     Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
      --no-default-features     Do not activate the `default` feature
  -j, --jobs <N>                Number of parallel jobs, defaults to # of CPUs
      --keep-going              Do not abort the build as soon as there is an error (unstable)
      --dry-run                 Perform all checks without uploading
      --registry <REGISTRY>     Registry to publish to
  -h, --help                    Print help information

Run `cargo help publish` for more detailed information.

Generally speaking, it's a good idea to perform some pre-flight checks before publishing a new crate version, so that you don't accidentally publish something in error. For instance, if there's a typo in the last small code change, and as a result, the crate no longer builds.

Variadic Command Parameters

First, let me explain a pattern I use in defining commands in a justfile.

Often, it's convenient to specify environment variables and other context in a justfile, so every command includes that environment context automatically.

This, in turn, makes it advantageous to execute commands via just, vs. executing the "bare" commands in a terminal. However, it's also necessary to be able to run unexpected, ad-hoc commands without incident. One possible problem, for example, is that cargo will re-compile everything whenever it is invoked with a different environment context, vs. performing its normal incremental build. If you had been invoking commands via just and then you use cargo directly in a terminal, the change in environment will trigger a full rebuild, which is annoying and time-consuming.

Variadic parameters are a useful tool for building flexible "wrapper" commands that allow you to execute common commands, including unexpected, ad-hoc commands, using just.

For example, consider this justfile:

# ~/src/my-rust-crate/justfile

set dotenv-load := true

export RUSTFLAGS := "-C target-cpu=native"
rustc-version := "nightly"

cargo +args='':
    cargo +{{rustc-version}} {{args}}

The +args='' is a variadic parameter, allowing any extra arguments to be passed at runtime, making it very flexible. E.g.:

$ just cargo check --example my-example

...is equivalent to running the command...

$ RUSTFLAGS='-C target-cpu=native' cargo +nightly check --example my-example

And you can combine several layers of these together:

# ~/src/my-rust-crate/justfile

set dotenv-load := true

export RUSTFLAGS := "-C target-cpu=native"
rustc-version := "nightly"

cargo +args='':
    cargo +{{rustc-version}} {{args}}

check +args='':
    @just cargo check {{args}}

debug-build binary-name +args='':
    @just cargo build --bin {{binary-name}} {{args}} 

release-build binary-name +args='':
    @just cargo build --bin {{binary-name}} --release {{args}}

prod-build binary-name +args='':
    @just release-build {{binary-name}} {{args}} --features prod

A few other miscellaneous notes:

• I have an alias j="just" in my ~/.bashrc so I can type j <cmd> (fewer letters == less typing == less time)
• set dotenv-load := true is now required (didn't used to be) for just to load the contents of .env at runtime
• once in a while I run into issues where I need to re-order or tweak how arguments and parameters are included in a command to prevent syntax errors in the post-interpolation rendering of it. For example, it may be necessary to use --features=prod instead of --features prod so the option is treated a single argument/string, preventing anything else from being inserted in between the --features and prod parts. Another example is changing the location of {{args}} in the command

Pre-Flight Checks

Now that we've got our just patterns down, here's how I apply them to the specific task of publishing a new crate version.

1. Pre-Release

pre-release:
    just cargo check \
        && just cargo test \
        && just cargo clippy \
        && echo "awesome job!"

This checks that

1. it builds
2. the tests pass
3. clippy isn't too angry (errors will trigger an error exit code, warnings won't)

Notes:

• You probably want to add && just cargo fmt to that list (for my own projects, no thanks, my code formatting is art tyvm)
• This could be a place where you check that builds work with several different configurations. For example run it with --features prod and --features dev or whatever.

2. Clean Working Directory: No Unstaged Changes

# somewhat hacky. works because the list of unstaged changes would be longer than 1 
verify-clean-git:
    test "$(echo `git status --porcelain` | wc -c)" -eq "1"

3. No Git Tag for Version Number Already Exists

I like to tag releases in git, which the publish command will cover.

This check also helps ensure that the version has been updated since the last time the crate was published. Otherwise, the tag would already exist.

A helper command to parse the version from Cargo.toml is included (get-crate-version).

get-crate-version:
    @cat Cargo.toml | rg '^version =' | sed -e 's/^version\s*=\s*//' | tr -d '"'

verify-release-tag-does-not-exist:
    VERSION=$(just get-crate-version) \
        && test -z "$(git tag | rg \"v${VERSION}\")" # Error: tag appears to exist already

4. Prepare Sqlx Offline Mode (Optional)

This only applies if you are using the sqlx crate as a dependency. This command is used to update the sqlx-data.json file that is used to run tests in environments where DATABASE_URL environment variable is not present (or SQLX_OFFLINE=true).

set dotenv-load := true
# note: DATABASE_URL (needed by sqlx command) defined in .env

# run sqlx prepare
prep-sqlx-offline:
    cargo install sqlx-cli
    cargo +{{rustc-version}} sqlx prepare -- --features prod --lib

# run sqlx prepare and then commit changes if there are any
update-sqlx-offline-and-commit:
    just prep-sqlx-offline
    just verify-clean-git \
        || ( sleep 0.25 && git add sqlx-data.json && git commit -m 'update sqlx-data.json')
    just verify-clean-git

Notes:

• The reason we always run cargo install sqlx-cli in this command is because the version of sqlx-cli needs to match the version of sqlx used by your crate. When sqlx is updated, the previous sqlx-cli prepare output (i.e. sqlx-data.json) will no longer be compatible. If the current version installed is the latest, this is a noop
• You may notice that the prep-sqlx-offline command did not utilize the just cargo {{args}} convention: the reason is that it needed to pass args to cargo after the command args, using the -- separator (i.e. cargo sqlx prepare <sqlx args> -- <cargo args>), so it wouldn't work here. That kind of thing happens, but it might be a tricky error when encountered

The Final Command

Putting it all together:

# ~/src/my-rust-crate/justfile

rustc-version := "nightly"
publish-registry := "shipyard-rs"
publish-features := "example-a example-b"

publish +args='': verify-clean-git verify-release-tag-does-not-exist pre-release
    just update-sqlx-offline-and-commit
    git push
    sleep 0.25
    cargo +{{rustc-version}} publish \
        --registry {{publish-registry}} \
        --features {{publish-features}} \
        --no-default-features {{args}}
    git tag "v$(just get-crate-version)"
    git push --tags
    rm -rf target/package

Notes:

• the commands listed to the right of the colon on the same line as publish are dependencies, similar to how a Makefile works, and will be run prior to execution of publish (and must complete successfully)
• to ensure it runs after the pre-flight checks are completed (the dependency commands), it was necessary to put just update-sqlx-offline-and-commit in the body of the command definition, vs. as a dependent command itself
• I added sleep 0.25 when getting it to work the first time, and now I can't remember why. But it's included here as example of possible mitigation tool when confronted with asynchronous, non-deterministic issues that crop up
• the git tag comes at the very end of the process, so that there's nothing else that could go wrong, since it's a bit annoying to need to delete it after the fact
• rm -rf target/package: as best I can tell, there is no advantage in terms of future incremental compilation to saving the contents of target/package directory generated by cargo publish, so I always delete it to save the disk space
• publishing to the private registry requires additional configuration in ~/.cargo/config.toml and ~/.cargo/credentials (for supplying the auth token), consult docs as needed

Self-Contained Example Justfile

set dotenv-load := true

rustc-version := "nightly"
publish-registry := "shipyard-rs"
publish-features := "prod"

export RUSTFLAGS := "-C link-arg=-fuse-ld=lld -C target-cpu=native"

# cargo [args].. (with env set)
cargo +args='':
    cargo +{{rustc-version}} {{args}}

# checks no unstaged changes
verify-clean-git:
    test "$(echo `git status --porcelain` | wc -c)" -eq "1"

# returns current version in Cargo.toml
get-crate-version:
    @cat Cargo.toml | rg '^version =' | sed -e 's/^version\s*=\s*//' | tr -d '"'

# ensures v{version} tag does not already exist
verify-release-tag-does-not-exist:
    VERSION=$(just get-crate-version) \
        && test -z "$(git tag | rg \"v${VERSION}\")" # Error: tag appears to exist already

# run sqlx prepare
prep-sqlx-offline:
    cargo install sqlx-cli
    cargo +{{rustc-version}} sqlx prepare -- --features {{publish-features}} --lib

# run sqlx prepare. if there are changes from that, commit them.
update-sqlx-offline-and-commit:
    just prep-sqlx-offline
    just verify-clean-git \
        || ( sleep 0.25 && git add sqlx-data.json && git commit -m 'update sqlx-data.json')
    just verify-clean-git

# ensure crate builds, tests pass, and clippy is not angry
pre-release:
    just cargo check \
        && just cargo test \
        && just cargo clippy \
        && echo "awesome job!"

# publish crate version to private registry
publish +args='': verify-clean-git verify-release-tag-does-not-exist pre-release
    just update-sqlx-offline-and-commit
    git push
    sleep 0.25
    cargo +{{rustc-version}} publish \
        --registry {{publish-registry}} \
        --features {{publish-features}} \
        --no-default-features {{args}}
    echo "adding git tag, now that EVERYTHING worked..."
    git tag "v$(just get-crate-version)"
    git push --tags
    rm -rf target/package