Quantum Experiment Reproducibility Checklist: Record Every Setting for Repeatable Results

Overview

Quantum experiment reproducibility means preserving enough information for someone else to rerun an experiment, understand any differences, and distinguish a genuine result from an environment or configuration change. The goal is not always to obtain identical measurement counts. Simulators can often be made highly repeatable, while hardware results may vary because of calibration changes, queue timing, device noise, and other execution conditions.

A useful experiment record therefore has two parts:

• Reconstruction data: the code, circuit, dependencies, inputs, backend, options, and random seeds required to repeat the run.
• Interpretation data: the expectations, metrics, plots, comparisons, exclusions, and conclusions used to explain the output.

Record the full context even when the experiment is small. A two-qubit demonstration can become a useful regression test, teaching example, or baseline for a larger quantum machine learning project. Before execution, you may also want to estimate qubit count, circuit depth, and gate cost using the quantum circuit resource estimation guide.

What to track

1. Experiment identity and objective

Start with a short record that explains what the experiment is intended to test. Include a unique experiment ID, date and time in a stated time zone, author, repository commit or archive identifier, and a one-sentence hypothesis. Define the primary outcome before looking at results. For example, the outcome might be an expectation value, success probability, classification score, or difference between an ideal and noisy simulation.

2. Software environment

Capture the complete execution environment rather than naming only the main SDK. Record:

• Programming language and version.
• Quantum SDK and package versions, including provider or plugin packages.
• Operating system and processor architecture when they may affect installation or simulation.
• Important numerical, visualization, machine learning, and data-processing dependencies.
• Environment file, lockfile, container digest, or other reproducible installation artifact.
• Relevant environment variables, configuration files, and provider settings, without storing secrets.

Package names and parameter names change over time. Copy the actual installed versions from the environment instead of relying on memory or a tutorial's original setup.

3. Circuit and input definition

Save the source code that constructs the circuit, but also preserve a machine-readable circuit representation when practical. Record the number of qubits and classical bits, register layout, gate sequence, parameter values, initialization state, measurement placement, and any mid-circuit operations or resets. If the circuit is generated dynamically, save the input data and generation configuration needed to create it again.

Document conventions that can silently change results, such as qubit ordering, bit-string ordering, endianness, basis-state notation, parameter units, and whether measurements are included during transpilation or added afterward.

4. Randomness and repeat count

List every random source used by the experiment. This may include circuit generation, parameter initialization, data shuffling, simulator sampling, transpiler choices, and optimization routines. Record the seed, seed-setting method, and number of repetitions. A single seed is not enough if several libraries have independent random-number generators.

For hardware experiments, separate the number of circuit executions, or shots, from the number of independent jobs or repeated submissions. These are different sources of variation and should be analyzed separately.

5. Transpilation and execution settings

Record the target backend or simulator, optimization level, basis gates or target constraints, coupling map when relevant, layout method, routing method, initial layout, scheduling options, dynamical decoupling settings, resilience or error-mitigation options, and any custom pass manager. Save both the original circuit and the final circuit sent for execution. The final circuit's depth, gate counts, and two-qubit gate counts are often essential for explaining a change in output.

For guidance on noisy-device results, connect this log to a separately documented quantum error mitigation workflow. Do not describe a result as unmitigated, mitigated, or error-corrected unless the record states exactly which processing was applied.

6. Backend, calibration, and noise information

For local simulation, record the simulator name, method, precision, device model, noise model, and any custom error parameters. For cloud hardware, record the provider, backend name, backend revision or identifier when available, job ID, submission and completion times, queue-related information if exposed, and the backend configuration used.

Capture calibration data or a reference to it when the platform makes it available. At minimum, note the calibration timestamp and relevant device status. A hardware result without its execution context may be difficult to compare with a later result, even when the circuit code is unchanged.

7. Shots, outputs, and analysis

Record the shot count for every circuit and control condition. Save raw measurement counts or bit strings before filtering, post-selection, mitigation, normalization, or plotting. Preserve intermediate files and the scripts that transform them into final figures or tables.

Describe statistical treatment clearly: discarded outcomes, aggregation rules, confidence intervals, repeated-run summaries, baseline selection, and rounding. State whether a result is an observed count, an estimated probability, an expectation value, or a model-derived quantity.

Compact experiment log template

Experiment ID:
Objective and hypothesis:
Author / repository / commit:
Date and time zone:
Language and SDK versions:
Dependency lockfile or container:
Circuit source and serialized circuit:
Qubits / classical bits / ordering convention:
Inputs and parameter values:
Random sources, seeds, and repetitions:
Backend or simulator:
Transpiler and execution settings:
Original and executed circuit metrics:
Noise model or calibration reference:
Shots per circuit / number of jobs:
Job IDs and execution timestamps:
Raw outputs location:
Analysis script and statistical method:
Primary result:
Limitations and unexplained differences:

Environment comparison

Cadence and checkpoints

Use checkpoints instead of waiting until the end of a project. At the design checkpoint, write the hypothesis, primary metric, baseline, circuit convention, and planned controls. At the environment checkpoint, freeze or record dependencies and run a small smoke test. At the execution checkpoint, capture the final circuit, backend details, seeds, shots, and job identifiers. At the analysis checkpoint, archive raw outputs and verify that the analysis script runs from a clean environment.

For collaborative work, review the log at least once per milestone or handoff. For a continuing hardware study, a monthly or quarterly review is useful: check whether SDK parameters, provider requirements, backend identifiers, calibration fields, or storage locations have changed. The exact interval should reflect how frequently the project executes and how quickly its dependencies change.

A practical repository layout might include src/ for circuit and analysis code, configs/ for declared settings, data/raw/ for untouched outputs, data/processed/ for derived files, logs/ for experiment records, and an environment lockfile at the project root.

How to interpret changes

When a replication differs, change one layer at a time. First verify the code commit, input data, package versions, circuit diagram, qubit ordering, and shot count. Next compare transpilation results: layout, routing, depth, and two-qubit gates. Then compare simulator method, precision, noise model, backend, calibration context, and mitigation settings.

Classify the difference rather than treating every mismatch as a failure:

• Deterministic mismatch: likely caused by code, dependency, circuit, input, or configuration differences.
• Sampling variation: plausible when finite shots produce different counts from the same probability distribution.
• Hardware variation: associated with backend conditions, calibration changes, scheduling, or device noise.
• Analysis mismatch: caused by filtering, aggregation, normalization, or metric definitions.

Compare distributions and uncertainty where appropriate instead of demanding identical bit-string counts. Preserve failed runs and unexpected outputs; they can reveal a hidden dependency or an assumption that the original record did not state.

When to revisit

Revisit this checklist whenever you start a new experiment, change an SDK or provider, move from simulation to hardware, alter transpiler settings, add error mitigation, change the circuit or dataset, or submit results for review. Also review it after a failed replication, a team handoff, or a long gap between executions.

Every six months is a reasonable maintenance reminder for an active project, with an earlier review after a major platform release or a change to provider access requirements. On each review, rerun a small reference circuit, verify that the environment can still be installed, check that stored job and calibration metadata remain readable, and update screenshots or parameter names that no longer match the working software.

Before publishing or archiving, use this final action list:

1. Run the experiment from a clean environment.
2. Confirm that the repository commit and environment record match the reported result.
3. Save the original circuit, executed circuit, raw outputs, and analysis artifacts.
4. Separate simulator assumptions from hardware observations.
5. State known limitations, nondeterministic components, and unavailable metadata.
6. Ask a colleague to reproduce one representative run using only the archived instructions.

For a broader learning path, the quantum computing roadmap can help place reproducibility practices alongside circuit design, simulation, and quantum machine learning skills. A well-maintained experiment log is not extra paperwork: it is the interface that lets future you, your collaborators, and independent readers understand what actually happened.