12:03:51 | INFO | __init__.py | JAX version 0.6.2 available. 12:03:57 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:03:58 | INFO | utils.py | Using cached dataset for librakevin/lewm-pusht at /root/.stable_worldmodel/datasets/librakevin--lewm-pusht [2026-10-10 12:03:58,358][root][WARNING] - LanceDataset: keys_to_cache=['action', 'proprio', 'state'] is not required — Lance has efficient random access via batched __getitems__. 12:04:11 | INFO | utils.py | Created ViT-tiny from scratch with config: {'hidden_size': 192, 'num_hidden_layers': 12, 'num_attention_heads': 3, 'intermediate_size': 768, 'image_size': 224, 'patch_size': 14} 12:04:11 | INFO | module.py | Setting up DataModule 12:04:11 | INFO | utils.py | ── Module ──────────────────────────────────────── 12:04:11 | WARN | module.py | ! No hyperparameters provided - hyperparameter logging is disabled. 12:04:11 | INFO | module.py | Setting custom forward method. 12:04:11 | INFO | module.py | Setting attribute: self.model = 12:04:11 | INFO | module.py | Setting attribute: self.sigreg = 12:04:11 | INFO | module.py | Setting attribute: self.optim = 12:04:11 | INFO | module.py | No metrics configuration provided - automatic metric tracking is disabled. /root/le-wm/.venv/lib/python3.10/site-packages/lightning/fabric/connector.py:571: `precision=bf16` is supported for historical reasons but its usage is discouraged. Please set your precision to bf16-mixed instead! /root/le-wm/.venv/lib/python3.10/site-packages/lightning/fabric/plugins/environments/slurm.py:204: The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: srun python train.py data=pusht img_size=224 trainer.max_epochs= ... Using bfloat16 Automatic Mixed Precision (AMP) 12:04:11 | INFO | env_info.py | EnvironmentDumpCallback initialized (filename=environment.json, async=True) 12:04:11 | INFO | utils.py | ── HuggingFaceCheckpoint ───────────────────────── 12:04:11 | INFO | hf_models.py| save_dir: hf_exports 12:04:11 | INFO | hf_models.py| per_step: False (False ⇒ overwrite single 'last/' subdir; True ⇒ keep step_N/) Adding 14 callbacks from entry point 'stablepretraining_callbacks': PrintProgressBar, ModuleRegistryCallback, LoggingCallback, EnvironmentDumpCallback, TrainerInfo, SklearnCheckpoint, WandbCheckpoint, TrackioCheckpoint, SwanLabCheckpoint, ModuleSummary, SLURMInfo, LogUnusedParametersOnce, HuggingFaceCheckpointCallback, HardwareMonitor GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/logger_connector/logger_connector.py:76: Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `lightning.pytorch` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default 💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform. 12:04:11 | INFO | utils.py | ── SIGTERM preempt handler — install ───────────── 12:04:11 | INFO | manager.py | SLURM_JOB_ID = None 12:04:11 | INFO | manager.py | SLURM_RESTART_COUNT = '0' 12:04:11 | INFO | manager.py | SLURM_PROCID = '?' 12:04:11 | INFO | manager.py | SLURM_NODEID = '?' 12:04:11 | INFO | manager.py | → not under SLURM (SLURM_JOB_ID unset); leaving SIGTERM handler as-is. Forwarding to submitit's USR signal would be a no-op since submitit isn't running upstream. 12:04:11 | WARN | manager.py | User didn't specify seed, runs won't be exactly reproducible! 12:04:11 | INFO | manager.py | Manager init: ckpt_path=None, weights_only=True 12:04:11 | INFO | utils.py | ── WorkingDirectory ────────────────────────────── 12:04:11 | INFO | manager.py | cwd: /root/le-wm 12:04:11 | INFO | utils.py | ── Seed ────────────────────────────────────────── 12:04:11 | INFO | manager.py | seed: None /root/le-wm/.venv/lib/python3.10/site-packages/stable_pretraining/manager.py:1871: UserWarning: No seed found, seed set to 0 seed_everything(self.seed, workers=True) 12:04:11 | INFO | distributed~| Seed set to 0 12:04:11 | INFO | utils.py | ── RunDirectory ────────────────────────────────── 12:04:11 | INFO | manager.py | cache_dir = /root/.cache/stable-pretraining 12:04:11 | INFO | manager.py | ddp: launch_key=(single-process) rank=0 is_rank_zero=True 12:04:11 | INFO | utils.py | ── RunDirectory: requeue probe ─────────────────── 12:04:11 | INFO | manager.py | SLURM session key = (no SLURM) 12:04:11 | INFO | manager.py | SLURM_RESTART_COUNT = 0 12:04:11 | INFO | manager.py | in_requeue = False 12:04:11 | INFO | utils.py | ── RunDirectory: fresh ─────────────────────────── 12:04:11 | INFO | manager.py | no user ckpt_path — training from scratch / pretrained backbone. 12:04:11 | INFO | manager.py | created run_dir = /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c 12:04:11 | INFO | manager.py | wrote run_meta.json with run_id=0a2e8732822c 12:04:11 | INFO | manager.py | SLURM index = (no SLURM, skipped) 12:04:11 | WARN | manager.py | ! cache_dir is set but the Trainer was passed as an already-instantiated object. Cannot override default_root_dir cleanly. Consider passing the trainer as a config dict instead. 12:04:11 | WARN | manager.py | ! Hydra run.dir='outputs/2026-10-10/12-03-58' will be ignored for trainer outputs (cache_dir takes precedence). 12:04:11 | WARN | manager.py | ! Hydra sweep.dir='multirun/2026-10-10/12-03-58' will be ignored for trainer outputs (cache_dir takes precedence). 12:04:11 | INFO | logger.py | [RegistryLogger] run_id=0a2e8732822c run_dir=/root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c — sidecar.json + summary.json + metrics.csv will live here 12:04:11 | INFO | utils.py | ── RegistryLogger ──────────────────────────────── 12:04:11 | INFO | manager.py | run_dir: /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c 12:04:11 | INFO | manager.py | run_id: 0a2e8732822c 12:04:11 | INFO | utils.py | ── CSVLogger ───────────────────────────────────── 12:04:11 | INFO | manager.py | root_dir: /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/ 12:04:11 | INFO | manager.py | save_dir: /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c 12:04:11 | INFO | manager.py | log_dir: /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/ 12:04:11 | INFO | utils.py | ── SignalHandlers (after submitit setup (spt SIGTERM installed in __init__)) ── 12:04:11 | INFO | manager.py | SIGUSR1 → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | SIGUSR2 → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | SIGCONT → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | SIGTERM → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | utils.py | ── Callbacks ───────────────────────────────────── 12:04:11 | INFO | manager.py | count: 20 12:04:11 | INFO | utils.py | ── CacheDirCheckpointing ───────────────────────── 12:04:11 | INFO | manager.py | Saving checkpoints to: /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints 12:04:11 | WARN | manager.py | Redirecting HuggingFaceCheckpointCallback from 'hf_exports' to '/root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/hf_exports' (cache_dir is active — prevents races on a shared cache) 12:04:11 | INFO | manager.py | Added requeue checkpoint (filename='last', every_n_train_steps=None) 12:04:11 | INFO | utils.py | ── LoadPath resolution ─────────────────────────── 12:04:11 | INFO | manager.py | in_requeue = False 12:04:11 | INFO | manager.py | user ckpt_path = None 12:04:11 | INFO | manager.py | user weights_only = True 12:04:11 | INFO | manager.py | → no ckpt_path; training from scratch. 12:04:11 | INFO | manager.py | manual-opt: moved gradient_clip_val=1.0 (algorithm=None) onto trainer.gradient_clip_val_; Module.training_step applies it via self.clip_gradients(). 12:04:11 | INFO | utils.py | ── TrainerFit ──────────────────────────────────── 12:04:11 | INFO | manager.py | ckpt_path: None 12:04:11 | INFO | manager.py | weights_only: None 12:04:11 | INFO | utils.py | ── SignalHandlers (before Trainer.fit() (handler installed in __init__)) ── 12:04:11 | INFO | manager.py | SIGUSR1 → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | SIGUSR2 → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | SIGCONT → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | SIGTERM → SIG_DFL (default OS action — terminate process) 12:04:11 | INFO | manager.py | → entering Trainer.fit(); lightning's _SignalConnector will now register its own handlers. SIGTERM will become _HandlersCompose([_sigterm_notifier_fn, _sigterm_handler_fn, spt._handler]); USR-sig binding from submitit is preserved (lightning skips USR registration when one already exists). 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. You are using a CUDA device ('NVIDIA GeForce RTX 5090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision 12:04:12 | OK | module.py | ✓ dataset for fit loaded 12:04:12 | INFO | module.py | setup was done by user 12:04:12 | INFO | utils.py | ── EnvironmentDump ─────────────────────────────── 12:04:12 | INFO | env_info.py | log_dir: /root/.cache/stable-pretraining 12:04:12 | INFO | env_info.py | Running environment dump in background thread (non-blocking) 12:04:12 | INFO | env_info.py | Collecting environment information... 12:04:12 | OK | env_info.py | ✓ Background thread started successfully 12:04:12 | OK | env_info.py | ✓ Python 3.10.18 12:04:12 | INFO | trainer_inf~| linking trainer to DataModule 12:04:12 | OK | env_info.py | ✓ System: Linux 6.8.0-139-generic (x86_64) 12:04:12 | INFO | utils.py | ── Callbacks (in order) ────────────────────────── 12:04:12 | INFO | env_info.py | Hostname: f9c895f69016 12:04:12 | INFO | utils.py | [ 0] SaveCkptCallback 12:04:12 | INFO | utils.py | [ 1] ResumeCkptCallback 12:04:12 | INFO | utils.py | [ 2] RichProgressBar 12:04:12 | INFO | utils.py | [ 3] RichModelSummary 12:04:12 | INFO | utils.py | [ 4] PrintProgressBar 12:04:12 | INFO | utils.py | [ 5] ModuleRegistryCallback 12:04:12 | INFO | utils.py | [ 6] LoggingCallback 12:04:12 | INFO | utils.py | [ 7] EnvironmentDumpCallback 12:04:12 | INFO | utils.py | [ 8] TrainerInfo 12:04:12 | INFO | utils.py | [ 9] SklearnCheckpoint 12:04:12 | INFO | utils.py | [10] WandbCheckpoint 12:04:12 | INFO | utils.py | [11] TrackioCheckpoint 12:04:12 | INFO | utils.py | [12] SwanLabCheckpoint 12:04:12 | INFO | utils.py | [13] ModuleSummary 12:04:12 | INFO | utils.py | [14] SLURMInfo 12:04:12 | INFO | utils.py | [15] LogUnusedParametersOnce 12:04:12 | INFO | utils.py | [16] HuggingFaceCheckpointCallback 12:04:12 | INFO | utils.py | [17] HardwareMonitor 12:04:12 | INFO | utils.py | [18] ModelCheckpoint 12:04:12 | INFO | utils.py | [19] _RunDirCallback 12:04:12 | INFO | utils.py | [20] ModelCheckpoint 12:04:12 | INFO | utils.py | ⚑ marks order-sensitive callbacks; see AGENTS.md → Callback ordering for the full rules. 12:04:12 | INFO | utils.py | ── SklearnCheckpoint ───────────────────────────── 12:04:12 | INFO | checkpoint_~| Sklearn Modules: 12:04:12 | INFO | checkpoint_~| ┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓ ┃ Module ┃ Name ┃ Type ┃ ┣━━━━━━━━━━╋━━━━━━━━╋━━━━━━━━┫ ┗━━━━━━━━━━┻━━━━━━━━┻━━━━━━━━┛ 12:04:12 | INFO | utils.py | ── WandbCheckpoint ─────────────────────────────── 12:04:12 | INFO | utils.py | ── TrackioCheckpoint ───────────────────────────── 12:04:12 | INFO | utils.py | ── SwanLabCheckpoint ───────────────────────────── 12:04:12 | INFO | utils.py | ── PyTorch Modules ─────────────────────────────── 12:04:12 | WARN | module.py | ! You are calling self.parameters which also gives callbacks parameters, to remove them, pass `with_callbacks=False` 12:04:12 | INFO | trainer_inf~| \n+---------------------------------------------------------+----------------------+--------------------------+--------------------------+---------+ | Module | Trainable parameters | Non Trainable parameters | Uninitialized parameters | Buffers | +---------------------------------------------------------+----------------------+--------------------------+--------------------------+---------+ | | 18034478 | 0 | 0 | 8245 | | callbacks_modules | 0 | 0 | 0 | 0 | | callbacks_metrics | 0 | 0 | 0 | 0 | | model | 18034478 | 0 | 0 | 8194 | | model.encoder | 5501376 | 0 | 0 | 0 | | model.encoder.embeddings | 162624 | 0 | 0 | 0 | | model.encoder.embeddings.patch_embeddings | 113088 | 0 | 0 | 0 | | model.encoder.embeddings.patch_embeddings.projection | 113088 | 0 | 0 | 0 | | model.encoder.embeddings.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers | 5338368 | 0 | 0 | 0 | | model.encoder.layers.0 | 444864 | 0 | 0 | 0 | | model.encoder.layers.0.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.0.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.0.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.0.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.0.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.0.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.0.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.0.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.0.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.0.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.0.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.0.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.1 | 444864 | 0 | 0 | 0 | | model.encoder.layers.1.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.1.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.1.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.1.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.1.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.1.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.1.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.1.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.1.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.1.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.1.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.1.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.2 | 444864 | 0 | 0 | 0 | | model.encoder.layers.2.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.2.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.2.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.2.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.2.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.2.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.2.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.2.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.2.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.2.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.2.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.2.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.3 | 444864 | 0 | 0 | 0 | | model.encoder.layers.3.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.3.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.3.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.3.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.3.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.3.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.3.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.3.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.3.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.3.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.3.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.3.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.4 | 444864 | 0 | 0 | 0 | | model.encoder.layers.4.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.4.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.4.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.4.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.4.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.4.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.4.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.4.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.4.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.4.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.4.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.4.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.5 | 444864 | 0 | 0 | 0 | | model.encoder.layers.5.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.5.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.5.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.5.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.5.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.5.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.5.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.5.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.5.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.5.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.5.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.5.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.6 | 444864 | 0 | 0 | 0 | | model.encoder.layers.6.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.6.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.6.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.6.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.6.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.6.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.6.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.6.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.6.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.6.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.6.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.6.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.7 | 444864 | 0 | 0 | 0 | | model.encoder.layers.7.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.7.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.7.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.7.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.7.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.7.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.7.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.7.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.7.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.7.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.7.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.7.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.8 | 444864 | 0 | 0 | 0 | | model.encoder.layers.8.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.8.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.8.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.8.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.8.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.8.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.8.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.8.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.8.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.8.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.8.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.8.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.9 | 444864 | 0 | 0 | 0 | | model.encoder.layers.9.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.9.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.9.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.9.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.9.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.9.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.9.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.9.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.9.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.9.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.9.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.9.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.10 | 444864 | 0 | 0 | 0 | | model.encoder.layers.10.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.10.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.10.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.10.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.10.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.10.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.10.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.10.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.10.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.10.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.10.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.10.dropout | 0 | 0 | 0 | 0 | | model.encoder.layers.11 | 444864 | 0 | 0 | 0 | | model.encoder.layers.11.attention | 148224 | 0 | 0 | 0 | | model.encoder.layers.11.attention.q_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.11.attention.k_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.11.attention.v_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.11.attention.o_proj | 37056 | 0 | 0 | 0 | | model.encoder.layers.11.layernorm_before | 384 | 0 | 0 | 0 | | model.encoder.layers.11.layernorm_after | 384 | 0 | 0 | 0 | | model.encoder.layers.11.mlp | 295872 | 0 | 0 | 0 | | model.encoder.layers.11.mlp.activation_fn | 0 | 0 | 0 | 0 | | model.encoder.layers.11.mlp.fc1 | 148224 | 0 | 0 | 0 | | model.encoder.layers.11.mlp.fc2 | 147648 | 0 | 0 | 0 | | model.encoder.layers.11.dropout | 0 | 0 | 0 | 0 | | model.encoder.layernorm | 384 | 0 | 0 | 0 | | model.predictor | 10791360 | 0 | 0 | 0 | | model.predictor.dropout | 0 | 0 | 0 | 0 | | model.predictor.transformer | 10790784 | 0 | 0 | 0 | | model.predictor.transformer.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers | 10790400 | 0 | 0 | 0 | | model.predictor.transformer.layers.0 | 1798400 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn | 787008 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn.attend | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn.to_qkv | 589824 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn.to_out | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn.to_out.0 | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.attn.to_out.1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net.0 | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net.1 | 395264 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net.2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net.3 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net.4 | 393408 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.mlp.net.5 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.norm1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.norm2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.adaLN_modulation | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.adaLN_modulation.0 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.0.adaLN_modulation.1 | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.1 | 1798400 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn | 787008 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn.attend | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn.to_qkv | 589824 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn.to_out | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn.to_out.0 | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.attn.to_out.1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net.0 | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net.1 | 395264 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net.2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net.3 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net.4 | 393408 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.mlp.net.5 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.norm1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.norm2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.adaLN_modulation | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.adaLN_modulation.0 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.1.adaLN_modulation.1 | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.2 | 1798400 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn | 787008 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn.attend | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn.to_qkv | 589824 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn.to_out | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn.to_out.0 | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.attn.to_out.1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net.0 | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net.1 | 395264 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net.2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net.3 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net.4 | 393408 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.mlp.net.5 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.norm1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.norm2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.adaLN_modulation | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.adaLN_modulation.0 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.2.adaLN_modulation.1 | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.3 | 1798400 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn | 787008 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn.attend | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn.to_qkv | 589824 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn.to_out | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn.to_out.0 | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.attn.to_out.1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net.0 | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net.1 | 395264 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net.2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net.3 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net.4 | 393408 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.mlp.net.5 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.norm1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.norm2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.adaLN_modulation | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.adaLN_modulation.0 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.3.adaLN_modulation.1 | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.4 | 1798400 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn | 787008 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn.attend | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn.to_qkv | 589824 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn.to_out | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn.to_out.0 | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.attn.to_out.1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net.0 | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net.1 | 395264 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net.2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net.3 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net.4 | 393408 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.mlp.net.5 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.norm1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.norm2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.adaLN_modulation | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.adaLN_modulation.0 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.4.adaLN_modulation.1 | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.5 | 1798400 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn | 787008 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn.norm | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn.attend | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn.to_qkv | 589824 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn.to_out | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn.to_out.0 | 196800 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.attn.to_out.1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net | 789056 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net.0 | 384 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net.1 | 395264 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net.2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net.3 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net.4 | 393408 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.mlp.net.5 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.norm1 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.norm2 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.adaLN_modulation | 222336 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.adaLN_modulation.0 | 0 | 0 | 0 | 0 | | model.predictor.transformer.layers.5.adaLN_modulation.1 | 222336 | 0 | 0 | 0 | | model.predictor.transformer.input_proj | 0 | 0 | 0 | 0 | | model.predictor.transformer.cond_proj | 0 | 0 | 0 | 0 | | model.predictor.transformer.output_proj | 0 | 0 | 0 | 0 | | model.action_encoder | 156206 | 0 | 0 | 0 | | model.action_encoder.patch_embed | 110 | 0 | 0 | 0 | | model.action_encoder.embed | 156096 | 0 | 0 | 0 | | model.action_encoder.embed.0 | 8448 | 0 | 0 | 0 | | model.action_encoder.embed.1 | 0 | 0 | 0 | 0 | | model.action_encoder.embed.2 | 147648 | 0 | 0 | 0 | | model.projector | 792768 | 0 | 0 | 4097 | | model.projector.net | 792768 | 0 | 0 | 4097 | | model.projector.net.0 | 395264 | 0 | 0 | 0 | | model.projector.net.1 | 4096 | 0 | 0 | 4097 | | model.projector.net.2 | 0 | 0 | 0 | 0 | | model.projector.net.3 | 393408 | 0 | 0 | 0 | | model.pred_proj | 792768 | 0 | 0 | 4097 | | model.pred_proj.net | 792768 | 0 | 0 | 4097 | | model.pred_proj.net.0 | 395264 | 0 | 0 | 0 | | model.pred_proj.net.1 | 4096 | 0 | 0 | 4097 | | model.pred_proj.net.2 | 0 | 0 | 0 | 0 | | model.pred_proj.net.3 | 393408 | 0 | 0 | 0 | | sigreg | 0 | 0 | 0 | 51 | +---------------------------------------------------------+----------------------+--------------------------+--------------------------+---------+ 12:04:12 | INFO | utils.py | ── SLURM ───────────────────────────────────────── 12:04:12 | INFO | trainer_inf~| Job ID: N/A 12:04:12 | INFO | trainer_inf~| Task ID: N/A 12:04:12 | INFO | trainer_inf~| Job Name: N/A 12:04:12 | INFO | trainer_inf~| Nodes: N/A 12:04:12 | INFO | trainer_inf~| Tasks: N/A 12:04:12 | INFO | trainer_inf~| CPUs per node: N/A 12:04:12 | INFO | trainer_inf~| CPUs per task: N/A 12:04:12 | INFO | trainer_inf~| Memory per node: N/A 12:04:12 | INFO | trainer_inf~| Memory per CPU: N/A 12:04:12 | INFO | trainer_inf~| Time limit: N/A 12:04:12 | INFO | trainer_inf~| Partition: N/A 12:04:12 | INFO | trainer_inf~| Node List: N/A 12:04:12 | INFO | trainer_inf~| Submit Directory: N/A 12:04:12 | INFO | utils.py | ── HardwareMonitor ─────────────────────────────── 12:04:12 | INFO | hardware_mo~| interval: 10.0s 12:04:12 | INFO | hardware_mo~| prefix: 'hardware/' 12:04:12 | INFO | hardware_mo~| metrics: cpu=True ram=True disk=True net=True gpu=True 12:04:12 | INFO | hardware_mo~| per-rank: rank 0 only LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] 12:04:12 | WARN | hardware_mo~| HardwareMonitor: pynvml unavailable (ModuleNotFoundError("No module named 'pynvml'")); GPU metrics disabled. 12:04:12 | WARN | env_info.py | ! pip freeze failed: Command '['/root/le-wm/.venv/bin/python', '-m', 'pip', 'freeze']' returned non-zero exit status 1. 12:04:12 | OK | env_info.py | ✓ Packages: 0 packages collected in 0.04s 12:04:12 | INFO | utils.py | ── Optimizers ──────────────────────────────────── 12:04:12 | INFO | module.py | Optimizer specified by Dict with keys ['model_opt'] 12:04:12 | INFO | module.py | ┏━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Optimizer ┃ Pattern ┃ Matched Modules ┃ Param Tensors ┃ Total Params ┃ RequiresGrad Tensors ┃ ┣━━━━━━━━━━━━━╋━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━┫ ┃ model_opt ┃ model ┃ 316 ┃ 297 ┃ 18034478 ┃ 297 ┃ ┗━━━━━━━━━━━━━┻━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━┛ 12:04:12 | INFO | lr_schedule~| User provided a dict Loading `train_dataloader` to estimate number of stepping batches. /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. 12:04:12 | OK | env_info.py | ✓ NVIDIA Driver: 595.91.07 12:04:13 | OK | env_info.py | ✓ Git: modified @ daa9793b 12:04:13 | INFO | env_info.py | Remote: https://github.com/taot/le-wm.git 12:04:13 | WARN | env_info.py | ! Working directory has uncommitted changes 12:04:13 | OK | env_info.py | ✓ Environment: 5 relevant variables captured 12:04:13 | INFO | env_info.py | Serializing environment data... 12:04:13 | INFO | env_info.py | Save directory: /root/.cache/stable-pretraining 12:04:13 | INFO | env_info.py | Writing environment to temp file... 12:04:13 | INFO | env_info.py | Moving to: environment.json 12:04:13 | OK | env_info.py | ✓ Environment file saved (1.5 KB) 12:04:13 | OK | env_info.py | ✓ Requirements file saved (0.1 KB) 12:04:20 | INFO | __init__.py | JAX version 0.6.2 available. 12:04:24 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:04:32 | INFO | __init__.py | JAX version 0.6.2 available. 12:04:36 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:04:45 | INFO | __init__.py | JAX version 0.6.2 available. 12:04:48 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:04:56 | INFO | __init__.py | JAX version 0.6.2 available. 12:04:59 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:05:08 | INFO | __init__.py | JAX version 0.6.2 available. 12:05:12 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:05:20 | INFO | __init__.py | JAX version 0.6.2 available. 12:05:24 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:05:24 | INFO | module.py | Configured optimizer 'model_opt' (modules=316, param_tensors=297, total_params=18034478) with LinearWarmupCosineAnnealingLR scheduler. /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/model_summary/model_summary.py:242: Precision bf16-mixed is not supported by the model summary. Estimated model size in MB will not be accurate. Using 32 bits instead. ┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━┩ │ 0 │ callbacks_modules │ ModuleDict │ 0 │ train │ 0 │ │ 1 │ callbacks_metrics │ ModuleDict │ 0 │ train │ 0 │ │ 2 │ model │ JEPA │ 18.0 M │ train │ 0 │ │ 3 │ sigreg │ SIGReg │ 0 │ train │ 0 │ └───┴───────────────────┴────────────┴────────┴───────┴───────┘ Trainable params: 18.0 M Non-trainable params: 0 Total params: 18.0 M Total estimated model params size (MB): 72.138 Modules in train mode: 319 Modules in eval mode: 0 Total FLOPs: 0 12:05:24 | INFO | logger.py | [RegistryLogger] summary.json flushed (0 metrics, step=0, epoch=0) → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/summary.json /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. 12:05:33 | INFO | __init__.py | JAX version 0.6.2 available. 12:05:37 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:05:46 | INFO | __init__.py | JAX version 0.6.2 available. 12:05:49 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:05:58 | INFO | __init__.py | JAX version 0.6.2 available. 12:06:01 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:06:09 | INFO | __init__.py | JAX version 0.6.2 available. 12:06:12 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:06:21 | INFO | __init__.py | JAX version 0.6.2 available. 12:06:25 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) 12:06:33 | INFO | __init__.py | JAX version 0.6.2 available. 12:06:37 | INFO | atomic_chec~| [atomic_save] installed crash-safe checkpoint plugin (write to sibling .tmp + fsync + atomic rename) /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:378: You have overridden `on_after_batch_transfer` in `LightningModule` but have passed in a `LightningDataModule`. It will use the implementation from `LightningModule` instance. /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/data.py:79: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 128. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`. 12:06:44 | INFO | trainer_inf~| \n+----------------------------+---------------------+ | Metric | Value | +----------------------------+---------------------+ | validate/loss | 5.010978698730469 | | validate/loss_epoch | 5.010978698730469 | | validate/pred_loss | 0.07347846031188965 | | validate/pred_loss_epoch | 0.07347846031188965 | | validate/sigreg_loss | 54.75 | | validate/sigreg_loss_epoch | 54.75 | +----------------------------+---------------------+ 12:06:44 | WARN | hf_models.py| ! No Hugging Face (PreTrainedModel) submodules found in LightningModule. 12:06:44 | INFO | utils.py | ── Optimizers ──────────────────────────────────── 12:06:44 | INFO | module.py | self.optimizers() gave us 1 optimizers 12:06:44 | WARN | module.py | ! No clip val found for optimizer model_opt 12:06:44 | WARN | module.py | ! Will use the Trainer's value of 1.0 12:06:44 | WARN | module.py | ! No clip algorithm found for optimizer model_opt 12:06:44 | WARN | module.py | ! Will use the Trainer's value of None 12:06:44 | OK | module.py | ✓ Optimizer check complete: +------------+-----------+----------------+-----------+-----------+ | Opt. Index | Opt. name | opt | clip val. | clip alg. | +------------+-----------+----------------+-----------+-----------+ | 0 | model_opt | LightningAdamW | 1.0 | None | +------------+-----------+----------------+-----------+-----------+ 12:06:44 | INFO | utils.py | ── LogUnusedParametersOnce ─────────────────────── 12:06:44 | INFO | unused_para~| registered hooks on 297 leaf parameters 12:06:45 | OK | unused_para~| ✓ all tracked parameters received gradients on the first backward pass 12:06:45 | INFO | unused_para~| hooks removed, callback disabled [Epoch 0/10] step 50/13933 (5.9 it/s) [Epoch 0/10] step 100/13933 (6.0 it/s) [Epoch 0/10] step 150/13933 (6.1 it/s) [Epoch 0/10] step 200/13933 (6.2 it/s) [Epoch 0/10] step 250/13933 (6.4 it/s) [Epoch 0/10] step 300/13933 (6.5 it/s) [Epoch 0/10] step 350/13933 (6.3 it/s) [Epoch 0/10] step 400/13933 (6.1 it/s) [Epoch 0/10] step 450/13933 (6.2 it/s) [Epoch 0/10] step 500/13933 (6.3 it/s) [Epoch 0/10] step 550/13933 (6.3 it/s) [Epoch 0/10] step 600/13933 (6.4 it/s) [Epoch 0/10] step 650/13933 (6.3 it/s) [Epoch 0/10] step 700/13933 (6.2 it/s) [Epoch 0/10] step 750/13933 (6.2 it/s) [Epoch 0/10] step 800/13933 (6.3 it/s) [Epoch 0/10] step 850/13933 (6.3 it/s) [Epoch 0/10] step 900/13933 (6.3 it/s) [Epoch 0/10] step 950/13933 (6.3 it/s) [Epoch 0/10] step 1000/13933 (6.3 it/s) [Epoch 0/10] step 1050/13933 (6.3 it/s) [Epoch 0/10] step 1100/13933 (6.3 it/s) [Epoch 0/10] step 1150/13933 (6.3 it/s) [Epoch 0/10] step 1200/13933 (6.3 it/s) [Epoch 0/10] step 1250/13933 (6.4 it/s) [Epoch 0/10] step 1300/13933 (6.3 it/s) [Epoch 0/10] step 1350/13933 (6.3 it/s) [Epoch 0/10] step 1400/13933 (6.3 it/s) [Epoch 0/10] step 1450/13933 (6.3 it/s) [Epoch 0/10] step 1500/13933 (6.4 it/s) [Epoch 0/10] step 1550/13933 (6.4 it/s) [Epoch 0/10] step 1600/13933 (6.4 it/s) [Epoch 0/10] step 1650/13933 (6.3 it/s) [Epoch 0/10] step 1700/13933 (6.4 it/s) [Epoch 0/10] step 1750/13933 (6.4 it/s) [Epoch 0/10] step 1800/13933 (6.4 it/s) [Epoch 0/10] step 1850/13933 (6.4 it/s) [Epoch 0/10] step 1900/13933 (6.4 it/s) [Epoch 0/10] step 1950/13933 (6.4 it/s) [Epoch 0/10] step 2000/13933 (6.4 it/s) [Epoch 0/10] step 2050/13933 (6.4 it/s) [Epoch 0/10] step 2100/13933 (6.4 it/s) [Epoch 0/10] step 2150/13933 (6.4 it/s) [Epoch 0/10] step 2200/13933 (6.4 it/s) [Epoch 0/10] step 2250/13933 (6.4 it/s) [Epoch 0/10] step 2300/13933 (6.4 it/s) [Epoch 0/10] step 2350/13933 (6.4 it/s) [Epoch 0/10] step 2400/13933 (6.4 it/s) [Epoch 0/10] step 2450/13933 (6.4 it/s) [Epoch 0/10] step 2500/13933 (6.4 it/s) [Epoch 0/10] step 2550/13933 (6.4 it/s) [Epoch 0/10] step 2600/13933 (6.4 it/s) [Epoch 0/10] step 2650/13933 (6.4 it/s) [Epoch 0/10] step 2700/13933 (6.4 it/s) [Epoch 0/10] step 2750/13933 (6.4 it/s) [Epoch 0/10] step 2800/13933 (6.4 it/s) [Epoch 0/10] step 2850/13933 (6.4 it/s) [Epoch 0/10] step 2900/13933 (6.4 it/s) [Epoch 0/10] step 2950/13933 (6.4 it/s) [Epoch 0/10] step 3000/13933 (6.4 it/s) [Epoch 0/10] step 3050/13933 (6.4 it/s) [Epoch 0/10] step 3100/13933 (6.5 it/s) [Epoch 0/10] step 3150/13933 (6.5 it/s) [Epoch 0/10] step 3200/13933 (6.5 it/s) [Epoch 0/10] step 3250/13933 (6.5 it/s) [Epoch 0/10] step 3300/13933 (6.5 it/s) [Epoch 0/10] step 3350/13933 (6.5 it/s) [Epoch 0/10] step 3400/13933 (6.5 it/s) [Epoch 0/10] step 3450/13933 (6.5 it/s) [Epoch 0/10] step 3500/13933 (6.5 it/s) [Epoch 0/10] step 3550/13933 (6.5 it/s) [Epoch 0/10] step 3600/13933 (6.5 it/s) [Epoch 0/10] step 3650/13933 (6.5 it/s) [Epoch 0/10] step 3700/13933 (6.5 it/s) [Epoch 0/10] step 3750/13933 (6.5 it/s) [Epoch 0/10] step 3800/13933 (6.5 it/s) [Epoch 0/10] step 3850/13933 (6.5 it/s) [Epoch 0/10] step 3900/13933 (6.5 it/s) [Epoch 0/10] step 3950/13933 (6.5 it/s) [Epoch 0/10] step 4000/13933 (6.5 it/s) [Epoch 0/10] step 4050/13933 (6.5 it/s) [Epoch 0/10] step 4100/13933 (6.5 it/s) [Epoch 0/10] step 4150/13933 (6.5 it/s) [Epoch 0/10] step 4200/13933 (6.5 it/s) [Epoch 0/10] step 4250/13933 (6.5 it/s) [Epoch 0/10] step 4300/13933 (6.5 it/s) [Epoch 0/10] step 4350/13933 (6.5 it/s) [Epoch 0/10] step 4400/13933 (6.5 it/s) [Epoch 0/10] step 4450/13933 (6.5 it/s) [Epoch 0/10] step 4500/13933 (6.5 it/s) [Epoch 0/10] step 4550/13933 (6.5 it/s) [Epoch 0/10] step 4600/13933 (6.5 it/s) [Epoch 0/10] step 4650/13933 (6.5 it/s) [Epoch 0/10] step 4700/13933 (6.5 it/s) [Epoch 0/10] step 4750/13933 (6.5 it/s) [Epoch 0/10] step 4800/13933 (6.5 it/s) [Epoch 0/10] step 4850/13933 (6.6 it/s) [Epoch 0/10] step 4900/13933 (6.6 it/s) [Epoch 0/10] step 4950/13933 (6.6 it/s) [Epoch 0/10] step 5000/13933 (6.6 it/s) [Epoch 0/10] step 5050/13933 (6.6 it/s) [Epoch 0/10] step 5100/13933 (6.6 it/s) [Epoch 0/10] step 5150/13933 (6.6 it/s) [Epoch 0/10] step 5200/13933 (6.6 it/s) [Epoch 0/10] step 5250/13933 (6.6 it/s) [Epoch 0/10] step 5300/13933 (6.6 it/s) [Epoch 0/10] step 5350/13933 (6.6 it/s) [Epoch 0/10] step 5400/13933 (6.6 it/s) [Epoch 0/10] step 5450/13933 (6.6 it/s) [Epoch 0/10] step 5500/13933 (6.6 it/s) [Epoch 0/10] step 5550/13933 (6.6 it/s) [Epoch 0/10] step 5600/13933 (6.6 it/s) [Epoch 0/10] step 5650/13933 (6.6 it/s) [Epoch 0/10] step 5700/13933 (6.6 it/s) [Epoch 0/10] step 5750/13933 (6.6 it/s) [Epoch 0/10] step 5800/13933 (6.6 it/s) [Epoch 0/10] step 5850/13933 (6.6 it/s) [Epoch 0/10] step 5900/13933 (6.6 it/s) [Epoch 0/10] step 5950/13933 (6.6 it/s) [Epoch 0/10] step 6000/13933 (6.6 it/s) [Epoch 0/10] step 6050/13933 (6.6 it/s) [Epoch 0/10] step 6100/13933 (6.6 it/s) [Epoch 0/10] step 6150/13933 (6.6 it/s) [Epoch 0/10] step 6200/13933 (6.6 it/s) [Epoch 0/10] step 6250/13933 (6.6 it/s) [Epoch 0/10] step 6300/13933 (6.6 it/s) [Epoch 0/10] step 6350/13933 (6.6 it/s) [Epoch 0/10] step 6400/13933 (6.6 it/s) [Epoch 0/10] step 6450/13933 (6.6 it/s) [Epoch 0/10] step 6500/13933 (6.6 it/s) [Epoch 0/10] step 6550/13933 (6.6 it/s) [Epoch 0/10] step 6600/13933 (6.6 it/s) [Epoch 0/10] step 6650/13933 (6.6 it/s) [Epoch 0/10] step 6700/13933 (6.6 it/s) [Epoch 0/10] step 6750/13933 (6.6 it/s) [Epoch 0/10] step 6800/13933 (6.6 it/s) [Epoch 0/10] step 6850/13933 (6.6 it/s) [Epoch 0/10] step 6900/13933 (6.6 it/s) [Epoch 0/10] step 6950/13933 (6.6 it/s) [Epoch 0/10] step 7000/13933 (6.6 it/s) [Epoch 0/10] step 7050/13933 (6.6 it/s) [Epoch 0/10] step 7100/13933 (6.6 it/s) [Epoch 0/10] step 7150/13933 (6.6 it/s) [Epoch 0/10] step 7200/13933 (6.6 it/s) [Epoch 0/10] step 7250/13933 (6.6 it/s) [Epoch 0/10] step 7300/13933 (6.6 it/s) [Epoch 0/10] step 7350/13933 (6.6 it/s) [Epoch 0/10] step 7400/13933 (6.6 it/s) [Epoch 0/10] step 7450/13933 (6.6 it/s) [Epoch 0/10] step 7500/13933 (6.6 it/s) [Epoch 0/10] step 7550/13933 (6.6 it/s) [Epoch 0/10] step 7600/13933 (6.6 it/s) [Epoch 0/10] step 7650/13933 (6.6 it/s) [Epoch 0/10] step 7700/13933 (6.6 it/s) [Epoch 0/10] step 7750/13933 (6.6 it/s) [Epoch 0/10] step 7800/13933 (6.6 it/s) [Epoch 0/10] step 7850/13933 (6.6 it/s) [Epoch 0/10] step 7900/13933 (6.6 it/s) [Epoch 0/10] step 7950/13933 (6.6 it/s) [Epoch 0/10] step 8000/13933 (6.6 it/s) [Epoch 0/10] step 8050/13933 (6.6 it/s) [Epoch 0/10] step 8100/13933 (6.6 it/s) [Epoch 0/10] step 8150/13933 (6.6 it/s) [Epoch 0/10] step 8200/13933 (6.6 it/s) [Epoch 0/10] step 8250/13933 (6.6 it/s) [Epoch 0/10] step 8300/13933 (6.6 it/s) [Epoch 0/10] step 8350/13933 (6.6 it/s) [Epoch 0/10] step 8400/13933 (6.6 it/s) [Epoch 0/10] step 8450/13933 (6.6 it/s) [Epoch 0/10] step 8500/13933 (6.6 it/s) [Epoch 0/10] step 8550/13933 (6.6 it/s) [Epoch 0/10] step 8600/13933 (6.6 it/s) [Epoch 0/10] step 8650/13933 (6.6 it/s) [Epoch 0/10] step 8700/13933 (6.6 it/s) [Epoch 0/10] step 8750/13933 (6.6 it/s) [Epoch 0/10] step 8800/13933 (6.6 it/s) [Epoch 0/10] step 8850/13933 (6.6 it/s) [Epoch 0/10] step 8900/13933 (6.6 it/s) [Epoch 0/10] step 8950/13933 (6.6 it/s) [Epoch 0/10] step 9000/13933 (6.6 it/s) [Epoch 0/10] step 9050/13933 (6.6 it/s) [Epoch 0/10] step 9100/13933 (6.6 it/s) [Epoch 0/10] step 9150/13933 (6.6 it/s) [Epoch 0/10] step 9200/13933 (6.6 it/s) [Epoch 0/10] step 9250/13933 (6.6 it/s) [Epoch 0/10] step 9300/13933 (6.6 it/s) [Epoch 0/10] step 9350/13933 (6.6 it/s) [Epoch 0/10] step 9400/13933 (6.6 it/s) [Epoch 0/10] step 9450/13933 (6.6 it/s) [Epoch 0/10] step 9500/13933 (6.6 it/s) [Epoch 0/10] step 9550/13933 (6.6 it/s) [Epoch 0/10] step 9600/13933 (6.6 it/s) [Epoch 0/10] step 9650/13933 (6.6 it/s) [Epoch 0/10] step 9700/13933 (6.6 it/s) [Epoch 0/10] step 9750/13933 (6.6 it/s) [Epoch 0/10] step 9800/13933 (6.6 it/s) [Epoch 0/10] step 9850/13933 (6.6 it/s) [Epoch 0/10] step 9900/13933 (6.6 it/s) [Epoch 0/10] step 9950/13933 (6.6 it/s) [Epoch 0/10] step 10000/13933 (6.6 it/s) [Epoch 0/10] step 10050/13933 (6.6 it/s) [Epoch 0/10] step 10100/13933 (6.6 it/s) [Epoch 0/10] step 10150/13933 (6.6 it/s) [Epoch 0/10] step 10200/13933 (6.6 it/s) [Epoch 0/10] step 10250/13933 (6.6 it/s) [Epoch 0/10] step 10300/13933 (6.6 it/s) [Epoch 0/10] step 10350/13933 (6.6 it/s) [Epoch 0/10] step 10400/13933 (6.6 it/s) [Epoch 0/10] step 10450/13933 (6.6 it/s) [Epoch 0/10] step 10500/13933 (6.6 it/s) [Epoch 0/10] step 10550/13933 (6.6 it/s) [Epoch 0/10] step 10600/13933 (6.6 it/s) [Epoch 0/10] step 10650/13933 (6.6 it/s) [Epoch 0/10] step 10700/13933 (6.6 it/s) [Epoch 0/10] step 10750/13933 (6.6 it/s) [Epoch 0/10] step 10800/13933 (6.6 it/s) [Epoch 0/10] step 10850/13933 (6.6 it/s) [Epoch 0/10] step 10900/13933 (6.6 it/s) [Epoch 0/10] step 10950/13933 (6.6 it/s) [Epoch 0/10] step 11000/13933 (6.6 it/s) [Epoch 0/10] step 11050/13933 (6.6 it/s) [Epoch 0/10] step 11100/13933 (6.6 it/s) [Epoch 0/10] step 11150/13933 (6.6 it/s) [Epoch 0/10] step 11200/13933 (6.6 it/s) [Epoch 0/10] step 11250/13933 (6.6 it/s) [Epoch 0/10] step 11300/13933 (6.6 it/s) [Epoch 0/10] step 11350/13933 (6.6 it/s) [Epoch 0/10] step 11400/13933 (6.6 it/s) [Epoch 0/10] step 11450/13933 (6.6 it/s) [Epoch 0/10] step 11500/13933 (6.6 it/s) [Epoch 0/10] step 11550/13933 (6.6 it/s) [Epoch 0/10] step 11600/13933 (6.6 it/s) [Epoch 0/10] step 11650/13933 (6.6 it/s) [Epoch 0/10] step 11700/13933 (6.6 it/s) [Epoch 0/10] step 11750/13933 (6.6 it/s) [Epoch 0/10] step 11800/13933 (6.6 it/s) [Epoch 0/10] step 11850/13933 (6.6 it/s) [Epoch 0/10] step 11900/13933 (6.6 it/s) [Epoch 0/10] step 11950/13933 (6.6 it/s) [Epoch 0/10] step 12000/13933 (6.6 it/s) [Epoch 0/10] step 12050/13933 (6.6 it/s) [Epoch 0/10] step 12100/13933 (6.6 it/s) [Epoch 0/10] step 12150/13933 (6.6 it/s) [Epoch 0/10] step 12200/13933 (6.6 it/s) [Epoch 0/10] step 12250/13933 (6.6 it/s) [Epoch 0/10] step 12300/13933 (6.6 it/s) [Epoch 0/10] step 12350/13933 (6.6 it/s) [Epoch 0/10] step 12400/13933 (6.6 it/s) [Epoch 0/10] step 12450/13933 (6.6 it/s) [Epoch 0/10] step 12500/13933 (6.6 it/s) [Epoch 0/10] step 12550/13933 (6.6 it/s) [Epoch 0/10] step 12600/13933 (6.6 it/s) [Epoch 0/10] step 12650/13933 (6.6 it/s) [Epoch 0/10] step 12700/13933 (6.6 it/s) [Epoch 0/10] step 12750/13933 (6.6 it/s) [Epoch 0/10] step 12800/13933 (6.6 it/s) [Epoch 0/10] step 12850/13933 (6.6 it/s) [Epoch 0/10] step 12900/13933 (6.6 it/s) [Epoch 0/10] step 12950/13933 (6.6 it/s) [Epoch 0/10] step 13000/13933 (6.6 it/s) [Epoch 0/10] step 13050/13933 (6.6 it/s) [Epoch 0/10] step 13100/13933 (6.6 it/s) [Epoch 0/10] step 13150/13933 (6.6 it/s) [Epoch 0/10] step 13200/13933 (6.6 it/s) [Epoch 0/10] step 13250/13933 (6.6 it/s) [Epoch 0/10] step 13300/13933 (6.6 it/s) [Epoch 0/10] step 13350/13933 (6.6 it/s) [Epoch 0/10] step 13400/13933 (6.6 it/s) [Epoch 0/10] step 13450/13933 (6.6 it/s) [Epoch 0/10] step 13500/13933 (6.6 it/s) [Epoch 0/10] step 13550/13933 (6.6 it/s) [Epoch 0/10] step 13600/13933 (6.6 it/s) [Epoch 0/10] step 13650/13933 (6.6 it/s) [Epoch 0/10] step 13700/13933 (6.6 it/s) [Epoch 0/10] step 13750/13933 (6.6 it/s) [Epoch 0/10] step 13800/13933 (6.6 it/s) [Epoch 0/10] step 13850/13933 (6.6 it/s) [Epoch 0/10] step 13900/13933 (6.6 it/s) /root/le-wm/.venv/lib/python3.10/site-packages/lightning/pytorch/utilities/data.py:79: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 28. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`. 12:46:25 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.1335747390985489 | | fit/pred_loss | 0.02371145784854889 | | fit/sigreg_loss | 1.21875 | | hardware/cpu_percent | 12.800000190734863 | | hardware/disk_read_mb_s | 1687.6922607421875 | | hardware/disk_write_mb_s | 0.20193414390087128 | | hardware/net_recv_mb_s | 0.00029439578065648675 | | hardware/net_sent_mb_s | 0.000844290480017662 | | hardware/ram_used_gb | 270.44451904296875 | | hardware/ram_used_pct | 53.70000076293945 | | hparams/lr_model_opt | 4.898728002444841e-05 | | validate/loss | 0.22852233052253723 | | validate/loss_epoch | 0.22852233052253723 | | validate/pred_loss | 0.024532422423362732 | | validate/pred_loss_epoch | 0.024532422423362732 | | validate/sigreg_loss | 2.2666213512420654 | | validate/sigreg_loss_epoch | 2.2666213512420654 | +----------------------------+------------------------+ 12:46:25 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_001.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 12:46:25 | INFO | checkpoint_~| Checking for non PyTorch modules to save 12:46:25 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 12:46:26 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.34s write=0.16s rename=0.1ms) [Epoch 0/10] done in 2381.5s 12:46:26 | INFO | checkpoint_~| Checking for non PyTorch modules to save 12:46:26 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=0-step=13933.ckpt 12:46:26 | INFO | atomic_chec~| [atomic_save] ✓ epoch=0-step=13933.ckpt saved (206.8 MiB, serialise=0.32s write=0.15s rename=0.1ms) 12:46:26 | INFO | checkpoint_~| Checking for non PyTorch modules to save 12:46:26 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 12:46:27 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.26s write=0.14s rename=0.1ms) [Epoch 1/10] step 50/13933 (4.5 it/s) [Epoch 1/10] step 100/13933 (4.8 it/s) [Epoch 1/10] step 150/13933 (5.4 it/s) [Epoch 1/10] step 200/13933 (5.7 it/s) [Epoch 1/10] step 250/13933 (5.9 it/s) [Epoch 1/10] step 300/13933 (6.0 it/s) [Epoch 1/10] step 350/13933 (6.0 it/s) [Epoch 1/10] step 400/13933 (5.8 it/s) [Epoch 1/10] step 450/13933 (5.8 it/s) [Epoch 1/10] step 500/13933 (5.7 it/s) [Epoch 1/10] step 550/13933 (5.8 it/s) [Epoch 1/10] step 600/13933 (5.9 it/s) [Epoch 1/10] step 650/13933 (6.0 it/s) [Epoch 1/10] step 700/13933 (6.0 it/s) [Epoch 1/10] step 750/13933 (6.0 it/s) [Epoch 1/10] step 800/13933 (6.0 it/s) [Epoch 1/10] step 850/13933 (6.0 it/s) [Epoch 1/10] step 900/13933 (5.9 it/s) [Epoch 1/10] step 950/13933 (5.9 it/s) [Epoch 1/10] step 1000/13933 (6.0 it/s) [Epoch 1/10] step 1050/13933 (6.0 it/s) [Epoch 1/10] step 1100/13933 (6.0 it/s) [Epoch 1/10] step 1150/13933 (6.1 it/s) [Epoch 1/10] step 1200/13933 (6.0 it/s) [Epoch 1/10] step 1250/13933 (6.0 it/s) [Epoch 1/10] step 1300/13933 (6.0 it/s) [Epoch 1/10] step 1350/13933 (6.0 it/s) [Epoch 1/10] step 1400/13933 (6.0 it/s) [Epoch 1/10] step 1450/13933 (6.1 it/s) [Epoch 1/10] step 1500/13933 (6.1 it/s) [Epoch 1/10] step 1550/13933 (6.0 it/s) [Epoch 1/10] step 1600/13933 (6.0 it/s) [Epoch 1/10] step 1650/13933 (6.1 it/s) [Epoch 1/10] step 1700/13933 (6.1 it/s) [Epoch 1/10] step 1750/13933 (6.1 it/s) [Epoch 1/10] step 1800/13933 (6.1 it/s) [Epoch 1/10] step 1850/13933 (6.1 it/s) [Epoch 1/10] step 1900/13933 (6.1 it/s) [Epoch 1/10] step 1950/13933 (6.1 it/s) [Epoch 1/10] step 2000/13933 (6.1 it/s) [Epoch 1/10] step 2050/13933 (6.1 it/s) [Epoch 1/10] step 2100/13933 (6.1 it/s) [Epoch 1/10] step 2150/13933 (6.2 it/s) [Epoch 1/10] step 2200/13933 (6.1 it/s) [Epoch 1/10] step 2250/13933 (6.1 it/s) [Epoch 1/10] step 2300/13933 (6.1 it/s) [Epoch 1/10] step 2350/13933 (6.1 it/s) [Epoch 1/10] step 2400/13933 (6.1 it/s) [Epoch 1/10] step 2450/13933 (6.2 it/s) [Epoch 1/10] step 2500/13933 (6.2 it/s) [Epoch 1/10] step 2550/13933 (6.1 it/s) [Epoch 1/10] step 2600/13933 (6.1 it/s) [Epoch 1/10] step 2650/13933 (6.1 it/s) [Epoch 1/10] step 2700/13933 (6.2 it/s) [Epoch 1/10] step 2750/13933 (6.2 it/s) [Epoch 1/10] step 2800/13933 (6.2 it/s) [Epoch 1/10] step 2850/13933 (6.2 it/s) [Epoch 1/10] step 2900/13933 (6.1 it/s) [Epoch 1/10] step 2950/13933 (6.2 it/s) [Epoch 1/10] step 3000/13933 (6.2 it/s) [Epoch 1/10] step 3050/13933 (6.2 it/s) [Epoch 1/10] step 3100/13933 (6.2 it/s) [Epoch 1/10] step 3150/13933 (6.2 it/s) [Epoch 1/10] step 3200/13933 (6.2 it/s) [Epoch 1/10] step 3250/13933 (6.2 it/s) [Epoch 1/10] step 3300/13933 (6.2 it/s) [Epoch 1/10] step 3350/13933 (6.2 it/s) [Epoch 1/10] step 3400/13933 (6.2 it/s) [Epoch 1/10] step 3450/13933 (6.2 it/s) [Epoch 1/10] step 3500/13933 (6.2 it/s) [Epoch 1/10] step 3550/13933 (6.2 it/s) [Epoch 1/10] step 3600/13933 (6.2 it/s) [Epoch 1/10] step 3650/13933 (6.2 it/s) [Epoch 1/10] step 3700/13933 (6.2 it/s) [Epoch 1/10] step 3750/13933 (6.2 it/s) [Epoch 1/10] step 3800/13933 (6.2 it/s) [Epoch 1/10] step 3850/13933 (6.2 it/s) [Epoch 1/10] step 3900/13933 (6.2 it/s) [Epoch 1/10] step 3950/13933 (6.2 it/s) [Epoch 1/10] step 4000/13933 (6.2 it/s) [Epoch 1/10] step 4050/13933 (6.2 it/s) [Epoch 1/10] step 4100/13933 (6.2 it/s) [Epoch 1/10] step 4150/13933 (6.2 it/s) [Epoch 1/10] step 4200/13933 (6.2 it/s) [Epoch 1/10] step 4250/13933 (6.2 it/s) [Epoch 1/10] step 4300/13933 (6.2 it/s) [Epoch 1/10] step 4350/13933 (6.2 it/s) [Epoch 1/10] step 4400/13933 (6.2 it/s) [Epoch 1/10] step 4450/13933 (6.2 it/s) [Epoch 1/10] step 4500/13933 (6.2 it/s) [Epoch 1/10] step 4550/13933 (6.2 it/s) [Epoch 1/10] step 4600/13933 (6.2 it/s) [Epoch 1/10] step 4650/13933 (6.2 it/s) [Epoch 1/10] step 4700/13933 (6.2 it/s) [Epoch 1/10] step 4750/13933 (6.2 it/s) [Epoch 1/10] step 4800/13933 (6.2 it/s) [Epoch 1/10] step 4850/13933 (6.2 it/s) [Epoch 1/10] step 4900/13933 (6.2 it/s) [Epoch 1/10] step 4950/13933 (6.2 it/s) [Epoch 1/10] step 5000/13933 (6.2 it/s) [Epoch 1/10] step 5050/13933 (6.2 it/s) [Epoch 1/10] step 5100/13933 (6.2 it/s) [Epoch 1/10] step 5150/13933 (6.2 it/s) [Epoch 1/10] step 5200/13933 (6.2 it/s) [Epoch 1/10] step 5250/13933 (6.2 it/s) [Epoch 1/10] step 5300/13933 (6.2 it/s) [Epoch 1/10] step 5350/13933 (6.2 it/s) [Epoch 1/10] step 5400/13933 (6.2 it/s) [Epoch 1/10] step 5450/13933 (6.2 it/s) [Epoch 1/10] step 5500/13933 (6.2 it/s) [Epoch 1/10] step 5550/13933 (6.2 it/s) [Epoch 1/10] step 5600/13933 (6.2 it/s) [Epoch 1/10] step 5650/13933 (6.2 it/s) [Epoch 1/10] step 5700/13933 (6.2 it/s) [Epoch 1/10] step 5750/13933 (6.2 it/s) [Epoch 1/10] step 5800/13933 (6.2 it/s) [Epoch 1/10] step 5850/13933 (6.2 it/s) [Epoch 1/10] step 5900/13933 (6.2 it/s) [Epoch 1/10] step 5950/13933 (6.2 it/s) [Epoch 1/10] step 6000/13933 (6.2 it/s) [Epoch 1/10] step 6050/13933 (6.2 it/s) [Epoch 1/10] step 6100/13933 (6.2 it/s) [Epoch 1/10] step 6150/13933 (6.2 it/s) [Epoch 1/10] step 6200/13933 (6.2 it/s) [Epoch 1/10] step 6250/13933 (6.2 it/s) [Epoch 1/10] step 6300/13933 (6.2 it/s) [Epoch 1/10] step 6350/13933 (6.2 it/s) [Epoch 1/10] step 6400/13933 (6.2 it/s) [Epoch 1/10] step 6450/13933 (6.2 it/s) [Epoch 1/10] step 6500/13933 (6.2 it/s) [Epoch 1/10] step 6550/13933 (6.2 it/s) [Epoch 1/10] step 6600/13933 (6.2 it/s) [Epoch 1/10] step 6650/13933 (6.2 it/s) [Epoch 1/10] step 6700/13933 (6.2 it/s) [Epoch 1/10] step 6750/13933 (6.2 it/s) [Epoch 1/10] step 6800/13933 (6.2 it/s) [Epoch 1/10] step 6850/13933 (6.2 it/s) [Epoch 1/10] step 6900/13933 (6.2 it/s) [Epoch 1/10] step 6950/13933 (6.2 it/s) [Epoch 1/10] step 7000/13933 (6.2 it/s) [Epoch 1/10] step 7050/13933 (6.2 it/s) [Epoch 1/10] step 7100/13933 (6.2 it/s) [Epoch 1/10] step 7150/13933 (6.2 it/s) [Epoch 1/10] step 7200/13933 (6.2 it/s) [Epoch 1/10] step 7250/13933 (6.2 it/s) [Epoch 1/10] step 7300/13933 (6.2 it/s) [Epoch 1/10] step 7350/13933 (6.2 it/s) [Epoch 1/10] step 7400/13933 (6.2 it/s) [Epoch 1/10] step 7450/13933 (6.2 it/s) [Epoch 1/10] step 7500/13933 (6.2 it/s) [Epoch 1/10] step 7550/13933 (6.2 it/s) [Epoch 1/10] step 7600/13933 (6.2 it/s) [Epoch 1/10] step 7650/13933 (6.2 it/s) [Epoch 1/10] step 7700/13933 (6.2 it/s) [Epoch 1/10] step 7750/13933 (6.2 it/s) [Epoch 1/10] step 7800/13933 (6.2 it/s) [Epoch 1/10] step 7850/13933 (6.2 it/s) [Epoch 1/10] step 7900/13933 (6.2 it/s) [Epoch 1/10] step 7950/13933 (6.2 it/s) [Epoch 1/10] step 8000/13933 (6.2 it/s) [Epoch 1/10] step 8050/13933 (6.2 it/s) [Epoch 1/10] step 8100/13933 (6.2 it/s) [Epoch 1/10] step 8150/13933 (6.2 it/s) [Epoch 1/10] step 8200/13933 (6.2 it/s) [Epoch 1/10] step 8250/13933 (6.2 it/s) [Epoch 1/10] step 8300/13933 (6.2 it/s) [Epoch 1/10] step 8350/13933 (6.2 it/s) [Epoch 1/10] step 8400/13933 (6.2 it/s) [Epoch 1/10] step 8450/13933 (6.2 it/s) [Epoch 1/10] step 8500/13933 (6.2 it/s) [Epoch 1/10] step 8550/13933 (6.2 it/s) [Epoch 1/10] step 8600/13933 (6.2 it/s) [Epoch 1/10] step 8650/13933 (6.2 it/s) [Epoch 1/10] step 8700/13933 (6.2 it/s) [Epoch 1/10] step 8750/13933 (6.2 it/s) [Epoch 1/10] step 8800/13933 (6.2 it/s) [Epoch 1/10] step 8850/13933 (6.2 it/s) [Epoch 1/10] step 8900/13933 (6.2 it/s) [Epoch 1/10] step 8950/13933 (6.2 it/s) [Epoch 1/10] step 9000/13933 (6.2 it/s) [Epoch 1/10] step 9050/13933 (6.2 it/s) [Epoch 1/10] step 9100/13933 (6.2 it/s) [Epoch 1/10] step 9150/13933 (6.2 it/s) [Epoch 1/10] step 9200/13933 (6.2 it/s) [Epoch 1/10] step 9250/13933 (6.2 it/s) [Epoch 1/10] step 9300/13933 (6.2 it/s) [Epoch 1/10] step 9350/13933 (6.2 it/s) [Epoch 1/10] step 9400/13933 (6.2 it/s) [Epoch 1/10] step 9450/13933 (6.2 it/s) [Epoch 1/10] step 9500/13933 (6.2 it/s) [Epoch 1/10] step 9550/13933 (6.2 it/s) [Epoch 1/10] step 9600/13933 (6.2 it/s) [Epoch 1/10] step 9650/13933 (6.2 it/s) [Epoch 1/10] step 9700/13933 (6.2 it/s) [Epoch 1/10] step 9750/13933 (6.2 it/s) [Epoch 1/10] step 9800/13933 (6.2 it/s) [Epoch 1/10] step 9850/13933 (6.2 it/s) [Epoch 1/10] step 9900/13933 (6.2 it/s) [Epoch 1/10] step 9950/13933 (6.2 it/s) [Epoch 1/10] step 10000/13933 (6.2 it/s) [Epoch 1/10] step 10050/13933 (6.2 it/s) [Epoch 1/10] step 10100/13933 (6.2 it/s) [Epoch 1/10] step 10150/13933 (6.2 it/s) [Epoch 1/10] step 10200/13933 (6.2 it/s) [Epoch 1/10] step 10250/13933 (6.2 it/s) [Epoch 1/10] step 10300/13933 (6.2 it/s) [Epoch 1/10] step 10350/13933 (6.2 it/s) [Epoch 1/10] step 10400/13933 (6.2 it/s) [Epoch 1/10] step 10450/13933 (6.2 it/s) [Epoch 1/10] step 10500/13933 (6.2 it/s) [Epoch 1/10] step 10550/13933 (6.2 it/s) [Epoch 1/10] step 10600/13933 (6.2 it/s) [Epoch 1/10] step 10650/13933 (6.2 it/s) [Epoch 1/10] step 10700/13933 (6.2 it/s) [Epoch 1/10] step 10750/13933 (6.2 it/s) [Epoch 1/10] step 10800/13933 (6.2 it/s) [Epoch 1/10] step 10850/13933 (6.2 it/s) [Epoch 1/10] step 10900/13933 (6.2 it/s) [Epoch 1/10] step 10950/13933 (6.2 it/s) [Epoch 1/10] step 11000/13933 (6.2 it/s) [Epoch 1/10] step 11050/13933 (6.2 it/s) [Epoch 1/10] step 11100/13933 (6.2 it/s) [Epoch 1/10] step 11150/13933 (6.2 it/s) [Epoch 1/10] step 11200/13933 (6.2 it/s) [Epoch 1/10] step 11250/13933 (6.2 it/s) [Epoch 1/10] step 11300/13933 (6.2 it/s) [Epoch 1/10] step 11350/13933 (6.2 it/s) [Epoch 1/10] step 11400/13933 (6.2 it/s) [Epoch 1/10] step 11450/13933 (6.2 it/s) [Epoch 1/10] step 11500/13933 (6.2 it/s) [Epoch 1/10] step 11550/13933 (6.2 it/s) [Epoch 1/10] step 11600/13933 (6.2 it/s) [Epoch 1/10] step 11650/13933 (6.2 it/s) [Epoch 1/10] step 11700/13933 (6.2 it/s) [Epoch 1/10] step 11750/13933 (6.2 it/s) [Epoch 1/10] step 11800/13933 (6.2 it/s) [Epoch 1/10] step 11850/13933 (6.2 it/s) [Epoch 1/10] step 11900/13933 (6.2 it/s) [Epoch 1/10] step 11950/13933 (6.2 it/s) [Epoch 1/10] step 12000/13933 (6.2 it/s) [Epoch 1/10] step 12050/13933 (6.2 it/s) [Epoch 1/10] step 12100/13933 (6.2 it/s) [Epoch 1/10] step 12150/13933 (6.2 it/s) [Epoch 1/10] step 12200/13933 (6.2 it/s) [Epoch 1/10] step 12250/13933 (6.2 it/s) [Epoch 1/10] step 12300/13933 (6.2 it/s) [Epoch 1/10] step 12350/13933 (6.2 it/s) [Epoch 1/10] step 12400/13933 (6.2 it/s) [Epoch 1/10] step 12450/13933 (6.2 it/s) [Epoch 1/10] step 12500/13933 (6.2 it/s) [Epoch 1/10] step 12550/13933 (6.2 it/s) [Epoch 1/10] step 12600/13933 (6.2 it/s) [Epoch 1/10] step 12650/13933 (6.2 it/s) [Epoch 1/10] step 12700/13933 (6.2 it/s) [Epoch 1/10] step 12750/13933 (6.2 it/s) [Epoch 1/10] step 12800/13933 (6.2 it/s) [Epoch 1/10] step 12850/13933 (6.2 it/s) [Epoch 1/10] step 12900/13933 (6.2 it/s) [Epoch 1/10] step 12950/13933 (6.2 it/s) [Epoch 1/10] step 13000/13933 (6.2 it/s) [Epoch 1/10] step 13050/13933 (6.2 it/s) [Epoch 1/10] step 13100/13933 (6.2 it/s) [Epoch 1/10] step 13150/13933 (6.2 it/s) [Epoch 1/10] step 13200/13933 (6.2 it/s) [Epoch 1/10] step 13250/13933 (6.2 it/s) [Epoch 1/10] step 13300/13933 (6.2 it/s) [Epoch 1/10] step 13350/13933 (6.2 it/s) [Epoch 1/10] step 13400/13933 (6.2 it/s) [Epoch 1/10] step 13450/13933 (6.2 it/s) [Epoch 1/10] step 13500/13933 (6.2 it/s) [Epoch 1/10] step 13550/13933 (6.2 it/s) [Epoch 1/10] step 13600/13933 (6.2 it/s) [Epoch 1/10] step 13650/13933 (6.2 it/s) [Epoch 1/10] step 13700/13933 (6.2 it/s) [Epoch 1/10] step 13750/13933 (6.2 it/s) [Epoch 1/10] step 13800/13933 (6.2 it/s) [Epoch 1/10] step 13850/13933 (6.2 it/s) [Epoch 1/10] step 13900/13933 (6.2 it/s) 13:27:54 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.11128818988800049 | | fit/pred_loss | 0.015585062094032764 | | fit/sigreg_loss | 1.0625 | | hardware/cpu_percent | 11.100000381469727 | | hardware/disk_read_mb_s | 1662.0341796875 | | hardware/disk_write_mb_s | 0.14903873205184937 | | hardware/net_recv_mb_s | 0.00014365297101903707 | | hardware/net_sent_mb_s | 0.00026421548682264984 | | hardware/ram_used_gb | 231.89915466308594 | | hardware/ram_used_pct | 46.099998474121094 | | hparams/lr_model_opt | 4.559183798846789e-05 | | validate/loss | 0.1621786504983902 | | validate/loss_epoch | 0.1621786504983902 | | validate/pred_loss | 0.01239214651286602 | | validate/pred_loss_epoch | 0.01239214651286602 | | validate/sigreg_loss | 1.6643575429916382 | | validate/sigreg_loss_epoch | 1.6643575429916382 | +----------------------------+------------------------+ 13:27:54 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_002.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 13:27:54 | INFO | checkpoint_~| Checking for non PyTorch modules to save 13:27:54 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 13:27:54 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.38s write=0.19s rename=0.3ms) [Epoch 1/10] done in 2487.8s 13:27:55 | INFO | checkpoint_~| Checking for non PyTorch modules to save 13:27:55 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=1-step=27866.ckpt 13:27:55 | INFO | atomic_chec~| [atomic_save] ✓ epoch=1-step=27866.ckpt saved (206.8 MiB, serialise=0.35s write=0.12s rename=0.1ms) 13:27:55 | INFO | checkpoint_~| Checking for non PyTorch modules to save 13:27:55 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 13:27:55 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.34s write=0.13s rename=0.1ms) [Epoch 2/10] step 50/13933 (5.9 it/s) [Epoch 2/10] step 100/13933 (6.5 it/s) [Epoch 2/10] step 150/13933 (6.6 it/s) [Epoch 2/10] step 200/13933 (6.7 it/s) [Epoch 2/10] step 250/13933 (6.6 it/s) [Epoch 2/10] step 300/13933 (6.4 it/s) [Epoch 2/10] step 350/13933 (6.4 it/s) [Epoch 2/10] step 400/13933 (6.5 it/s) [Epoch 2/10] step 450/13933 (6.5 it/s) [Epoch 2/10] step 500/13933 (6.5 it/s) [Epoch 2/10] step 550/13933 (6.6 it/s) [Epoch 2/10] step 600/13933 (6.5 it/s) [Epoch 2/10] step 650/13933 (6.4 it/s) [Epoch 2/10] step 700/13933 (6.5 it/s) [Epoch 2/10] step 750/13933 (6.5 it/s) [Epoch 2/10] step 800/13933 (6.5 it/s) [Epoch 2/10] step 850/13933 (6.5 it/s) [Epoch 2/10] step 900/13933 (6.5 it/s) [Epoch 2/10] step 950/13933 (6.4 it/s) [Epoch 2/10] step 1000/13933 (6.4 it/s) [Epoch 2/10] step 1050/13933 (6.4 it/s) [Epoch 2/10] step 1100/13933 (6.4 it/s) [Epoch 2/10] step 1150/13933 (6.5 it/s) [Epoch 2/10] step 1200/13933 (6.5 it/s) [Epoch 2/10] step 1250/13933 (6.4 it/s) [Epoch 2/10] step 1300/13933 (6.3 it/s) [Epoch 2/10] step 1350/13933 (6.4 it/s) [Epoch 2/10] step 1400/13933 (6.4 it/s) [Epoch 2/10] step 1450/13933 (6.4 it/s) [Epoch 2/10] step 1500/13933 (6.4 it/s) [Epoch 2/10] step 1550/13933 (6.4 it/s) [Epoch 2/10] step 1600/13933 (6.3 it/s) [Epoch 2/10] step 1650/13933 (6.4 it/s) [Epoch 2/10] step 1700/13933 (6.4 it/s) [Epoch 2/10] step 1750/13933 (6.4 it/s) [Epoch 2/10] step 1800/13933 (6.4 it/s) [Epoch 2/10] step 1850/13933 (6.4 it/s) [Epoch 2/10] step 1900/13933 (6.3 it/s) [Epoch 2/10] step 1950/13933 (6.3 it/s) [Epoch 2/10] step 2000/13933 (6.3 it/s) [Epoch 2/10] step 2050/13933 (6.4 it/s) [Epoch 2/10] step 2100/13933 (6.4 it/s) [Epoch 2/10] step 2150/13933 (6.4 it/s) [Epoch 2/10] step 2200/13933 (6.4 it/s) [Epoch 2/10] step 2250/13933 (6.4 it/s) [Epoch 2/10] step 2300/13933 (6.4 it/s) [Epoch 2/10] step 2350/13933 (6.4 it/s) [Epoch 2/10] step 2400/13933 (6.3 it/s) [Epoch 2/10] step 2450/13933 (6.4 it/s) [Epoch 2/10] step 2500/13933 (6.3 it/s) [Epoch 2/10] step 2550/13933 (6.3 it/s) [Epoch 2/10] step 2600/13933 (6.2 it/s) [Epoch 2/10] step 2650/13933 (6.2 it/s) [Epoch 2/10] step 2700/13933 (6.2 it/s) [Epoch 2/10] step 2750/13933 (6.2 it/s) [Epoch 2/10] step 2800/13933 (6.2 it/s) [Epoch 2/10] step 2850/13933 (6.3 it/s) [Epoch 2/10] step 2900/13933 (6.2 it/s) [Epoch 2/10] step 2950/13933 (6.2 it/s) [Epoch 2/10] step 3000/13933 (6.2 it/s) [Epoch 2/10] step 3050/13933 (6.2 it/s) [Epoch 2/10] step 3100/13933 (6.2 it/s) [Epoch 2/10] step 3150/13933 (6.2 it/s) [Epoch 2/10] step 3200/13933 (6.2 it/s) [Epoch 2/10] step 3250/13933 (6.2 it/s) [Epoch 2/10] step 3300/13933 (6.2 it/s) [Epoch 2/10] step 3350/13933 (6.1 it/s) [Epoch 2/10] step 3400/13933 (6.1 it/s) [Epoch 2/10] step 3450/13933 (6.2 it/s) [Epoch 2/10] step 3500/13933 (6.2 it/s) [Epoch 2/10] step 3550/13933 (6.2 it/s) [Epoch 2/10] step 3600/13933 (6.2 it/s) [Epoch 2/10] step 3650/13933 (6.1 it/s) [Epoch 2/10] step 3700/13933 (6.1 it/s) [Epoch 2/10] step 3750/13933 (6.2 it/s) [Epoch 2/10] step 3800/13933 (6.2 it/s) [Epoch 2/10] step 3850/13933 (6.2 it/s) [Epoch 2/10] step 3900/13933 (6.2 it/s) [Epoch 2/10] step 3950/13933 (6.2 it/s) [Epoch 2/10] step 4000/13933 (6.1 it/s) [Epoch 2/10] step 4050/13933 (6.2 it/s) [Epoch 2/10] step 4100/13933 (6.2 it/s) [Epoch 2/10] step 4150/13933 (6.2 it/s) [Epoch 2/10] step 4200/13933 (6.2 it/s) [Epoch 2/10] step 4250/13933 (6.2 it/s) [Epoch 2/10] step 4300/13933 (6.2 it/s) [Epoch 2/10] step 4350/13933 (6.2 it/s) [Epoch 2/10] step 4400/13933 (6.2 it/s) [Epoch 2/10] step 4450/13933 (6.2 it/s) [Epoch 2/10] step 4500/13933 (6.2 it/s) [Epoch 2/10] step 4550/13933 (6.2 it/s) [Epoch 2/10] step 4600/13933 (6.2 it/s) [Epoch 2/10] step 4650/13933 (6.2 it/s) [Epoch 2/10] step 4700/13933 (6.2 it/s) [Epoch 2/10] step 4750/13933 (6.3 it/s) [Epoch 2/10] step 4800/13933 (6.3 it/s) [Epoch 2/10] step 4850/13933 (6.3 it/s) [Epoch 2/10] step 4900/13933 (6.3 it/s) [Epoch 2/10] step 4950/13933 (6.3 it/s) [Epoch 2/10] step 5000/13933 (6.3 it/s) [Epoch 2/10] step 5050/13933 (6.3 it/s) [Epoch 2/10] step 5100/13933 (6.3 it/s) [Epoch 2/10] step 5150/13933 (6.3 it/s) [Epoch 2/10] step 5200/13933 (6.3 it/s) [Epoch 2/10] step 5250/13933 (6.3 it/s) [Epoch 2/10] step 5300/13933 (6.3 it/s) [Epoch 2/10] step 5350/13933 (6.3 it/s) [Epoch 2/10] step 5400/13933 (6.3 it/s) [Epoch 2/10] step 5450/13933 (6.3 it/s) [Epoch 2/10] step 5500/13933 (6.3 it/s) [Epoch 2/10] step 5550/13933 (6.4 it/s) [Epoch 2/10] step 5600/13933 (6.4 it/s) [Epoch 2/10] step 5650/13933 (6.4 it/s) [Epoch 2/10] step 5700/13933 (6.4 it/s) [Epoch 2/10] step 5750/13933 (6.4 it/s) [Epoch 2/10] step 5800/13933 (6.4 it/s) [Epoch 2/10] step 5850/13933 (6.4 it/s) [Epoch 2/10] step 5900/13933 (6.4 it/s) [Epoch 2/10] step 5950/13933 (6.4 it/s) [Epoch 2/10] step 6000/13933 (6.4 it/s) [Epoch 2/10] step 6050/13933 (6.4 it/s) [Epoch 2/10] step 6100/13933 (6.4 it/s) [Epoch 2/10] step 6150/13933 (6.4 it/s) [Epoch 2/10] step 6200/13933 (6.4 it/s) [Epoch 2/10] step 6250/13933 (6.4 it/s) [Epoch 2/10] step 6300/13933 (6.4 it/s) [Epoch 2/10] step 6350/13933 (6.4 it/s) [Epoch 2/10] step 6400/13933 (6.4 it/s) [Epoch 2/10] step 6450/13933 (6.4 it/s) [Epoch 2/10] step 6500/13933 (6.4 it/s) [Epoch 2/10] step 6550/13933 (6.4 it/s) [Epoch 2/10] step 6600/13933 (6.5 it/s) [Epoch 2/10] step 6650/13933 (6.5 it/s) [Epoch 2/10] step 6700/13933 (6.5 it/s) [Epoch 2/10] step 6750/13933 (6.5 it/s) [Epoch 2/10] step 6800/13933 (6.5 it/s) [Epoch 2/10] step 6850/13933 (6.5 it/s) [Epoch 2/10] step 6900/13933 (6.5 it/s) [Epoch 2/10] step 6950/13933 (6.5 it/s) [Epoch 2/10] step 7000/13933 (6.5 it/s) [Epoch 2/10] step 7050/13933 (6.5 it/s) [Epoch 2/10] step 7100/13933 (6.5 it/s) [Epoch 2/10] step 7150/13933 (6.5 it/s) [Epoch 2/10] step 7200/13933 (6.5 it/s) [Epoch 2/10] step 7250/13933 (6.5 it/s) [Epoch 2/10] step 7300/13933 (6.5 it/s) [Epoch 2/10] step 7350/13933 (6.5 it/s) [Epoch 2/10] step 7400/13933 (6.5 it/s) [Epoch 2/10] step 7450/13933 (6.5 it/s) [Epoch 2/10] step 7500/13933 (6.5 it/s) [Epoch 2/10] step 7550/13933 (6.5 it/s) [Epoch 2/10] step 7600/13933 (6.5 it/s) [Epoch 2/10] step 7650/13933 (6.5 it/s) [Epoch 2/10] step 7700/13933 (6.5 it/s) [Epoch 2/10] step 7750/13933 (6.5 it/s) [Epoch 2/10] step 7800/13933 (6.5 it/s) [Epoch 2/10] step 7850/13933 (6.5 it/s) [Epoch 2/10] step 7900/13933 (6.5 it/s) [Epoch 2/10] step 7950/13933 (6.5 it/s) [Epoch 2/10] step 8000/13933 (6.5 it/s) [Epoch 2/10] step 8050/13933 (6.5 it/s) [Epoch 2/10] step 8100/13933 (6.5 it/s) [Epoch 2/10] step 8150/13933 (6.5 it/s) [Epoch 2/10] step 8200/13933 (6.5 it/s) [Epoch 2/10] step 8250/13933 (6.5 it/s) [Epoch 2/10] step 8300/13933 (6.6 it/s) [Epoch 2/10] step 8350/13933 (6.6 it/s) [Epoch 2/10] step 8400/13933 (6.6 it/s) [Epoch 2/10] step 8450/13933 (6.6 it/s) [Epoch 2/10] step 8500/13933 (6.6 it/s) [Epoch 2/10] step 8550/13933 (6.6 it/s) [Epoch 2/10] step 8600/13933 (6.6 it/s) [Epoch 2/10] step 8650/13933 (6.6 it/s) [Epoch 2/10] step 8700/13933 (6.6 it/s) [Epoch 2/10] step 8750/13933 (6.6 it/s) [Epoch 2/10] step 8800/13933 (6.6 it/s) [Epoch 2/10] step 8850/13933 (6.6 it/s) [Epoch 2/10] step 8900/13933 (6.6 it/s) [Epoch 2/10] step 8950/13933 (6.6 it/s) [Epoch 2/10] step 9000/13933 (6.6 it/s) [Epoch 2/10] step 9050/13933 (6.6 it/s) [Epoch 2/10] step 9100/13933 (6.6 it/s) [Epoch 2/10] step 9150/13933 (6.6 it/s) [Epoch 2/10] step 9200/13933 (6.6 it/s) [Epoch 2/10] step 9250/13933 (6.6 it/s) [Epoch 2/10] step 9300/13933 (6.6 it/s) [Epoch 2/10] step 9350/13933 (6.6 it/s) [Epoch 2/10] step 9400/13933 (6.6 it/s) [Epoch 2/10] step 9450/13933 (6.6 it/s) [Epoch 2/10] step 9500/13933 (6.6 it/s) [Epoch 2/10] step 9550/13933 (6.6 it/s) [Epoch 2/10] step 9600/13933 (6.6 it/s) [Epoch 2/10] step 9650/13933 (6.6 it/s) [Epoch 2/10] step 9700/13933 (6.6 it/s) [Epoch 2/10] step 9750/13933 (6.6 it/s) [Epoch 2/10] step 9800/13933 (6.6 it/s) [Epoch 2/10] step 9850/13933 (6.6 it/s) [Epoch 2/10] step 9900/13933 (6.6 it/s) [Epoch 2/10] step 9950/13933 (6.6 it/s) [Epoch 2/10] step 10000/13933 (6.6 it/s) [Epoch 2/10] step 10050/13933 (6.6 it/s) [Epoch 2/10] step 10100/13933 (6.6 it/s) [Epoch 2/10] step 10150/13933 (6.6 it/s) [Epoch 2/10] step 10200/13933 (6.6 it/s) [Epoch 2/10] step 10250/13933 (6.6 it/s) [Epoch 2/10] step 10300/13933 (6.6 it/s) [Epoch 2/10] step 10350/13933 (6.6 it/s) [Epoch 2/10] step 10400/13933 (6.6 it/s) [Epoch 2/10] step 10450/13933 (6.6 it/s) [Epoch 2/10] step 10500/13933 (6.6 it/s) [Epoch 2/10] step 10550/13933 (6.6 it/s) [Epoch 2/10] step 10600/13933 (6.6 it/s) [Epoch 2/10] step 10650/13933 (6.6 it/s) [Epoch 2/10] step 10700/13933 (6.6 it/s) [Epoch 2/10] step 10750/13933 (6.6 it/s) [Epoch 2/10] step 10800/13933 (6.6 it/s) [Epoch 2/10] step 10850/13933 (6.6 it/s) [Epoch 2/10] step 10900/13933 (6.6 it/s) [Epoch 2/10] step 10950/13933 (6.6 it/s) [Epoch 2/10] step 11000/13933 (6.6 it/s) [Epoch 2/10] step 11050/13933 (6.6 it/s) [Epoch 2/10] step 11100/13933 (6.6 it/s) [Epoch 2/10] step 11150/13933 (6.6 it/s) [Epoch 2/10] step 11200/13933 (6.6 it/s) [Epoch 2/10] step 11250/13933 (6.6 it/s) [Epoch 2/10] step 11300/13933 (6.6 it/s) [Epoch 2/10] step 11350/13933 (6.6 it/s) [Epoch 2/10] step 11400/13933 (6.6 it/s) [Epoch 2/10] step 11450/13933 (6.6 it/s) [Epoch 2/10] step 11500/13933 (6.6 it/s) [Epoch 2/10] step 11550/13933 (6.6 it/s) [Epoch 2/10] step 11600/13933 (6.6 it/s) [Epoch 2/10] step 11650/13933 (6.6 it/s) [Epoch 2/10] step 11700/13933 (6.6 it/s) [Epoch 2/10] step 11750/13933 (6.6 it/s) [Epoch 2/10] step 11800/13933 (6.6 it/s) [Epoch 2/10] step 11850/13933 (6.6 it/s) [Epoch 2/10] step 11900/13933 (6.6 it/s) [Epoch 2/10] step 11950/13933 (6.6 it/s) [Epoch 2/10] step 12000/13933 (6.6 it/s) [Epoch 2/10] step 12050/13933 (6.6 it/s) [Epoch 2/10] step 12100/13933 (6.6 it/s) [Epoch 2/10] step 12150/13933 (6.6 it/s) [Epoch 2/10] step 12200/13933 (6.6 it/s) [Epoch 2/10] step 12250/13933 (6.6 it/s) [Epoch 2/10] step 12300/13933 (6.6 it/s) [Epoch 2/10] step 12350/13933 (6.6 it/s) [Epoch 2/10] step 12400/13933 (6.6 it/s) [Epoch 2/10] step 12450/13933 (6.6 it/s) [Epoch 2/10] step 12500/13933 (6.6 it/s) [Epoch 2/10] step 12550/13933 (6.6 it/s) [Epoch 2/10] step 12600/13933 (6.6 it/s) [Epoch 2/10] step 12650/13933 (6.6 it/s) [Epoch 2/10] step 12700/13933 (6.6 it/s) [Epoch 2/10] step 12750/13933 (6.6 it/s) [Epoch 2/10] step 12800/13933 (6.6 it/s) [Epoch 2/10] step 12850/13933 (6.6 it/s) [Epoch 2/10] step 12900/13933 (6.6 it/s) [Epoch 2/10] step 12950/13933 (6.6 it/s) [Epoch 2/10] step 13000/13933 (6.6 it/s) [Epoch 2/10] step 13050/13933 (6.6 it/s) [Epoch 2/10] step 13100/13933 (6.6 it/s) [Epoch 2/10] step 13150/13933 (6.6 it/s) [Epoch 2/10] step 13200/13933 (6.6 it/s) [Epoch 2/10] step 13250/13933 (6.6 it/s) [Epoch 2/10] step 13300/13933 (6.6 it/s) [Epoch 2/10] step 13350/13933 (6.6 it/s) [Epoch 2/10] step 13400/13933 (6.6 it/s) [Epoch 2/10] step 13450/13933 (6.6 it/s) [Epoch 2/10] step 13500/13933 (6.6 it/s) [Epoch 2/10] step 13550/13933 (6.6 it/s) [Epoch 2/10] step 13600/13933 (6.6 it/s) [Epoch 2/10] step 13650/13933 (6.6 it/s) [Epoch 2/10] step 13700/13933 (6.6 it/s) [Epoch 2/10] step 13750/13933 (6.6 it/s) [Epoch 2/10] step 13800/13933 (6.6 it/s) [Epoch 2/10] step 13850/13933 (6.6 it/s) [Epoch 2/10] step 13900/13933 (6.6 it/s) 14:06:31 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.10833107680082321 | | fit/pred_loss | 0.012139667756855488 | | fit/sigreg_loss | 1.0703125 | | hardware/cpu_percent | 18.700000762939453 | | hardware/disk_read_mb_s | 1753.39599609375 | | hardware/disk_write_mb_s | 201.69473266601562 | | hardware/net_recv_mb_s | 0.0001458753104088828 | | hardware/net_sent_mb_s | 0.00025933387223631144 | | hardware/ram_used_gb | 215.3981170654297 | | hardware/ram_used_pct | 42.79999923706055 | | hparams/lr_model_opt | 4.01401448471006e-05 | | validate/loss | 0.1478964388370514 | | validate/loss_epoch | 0.1478964388370514 | | validate/pred_loss | 0.008976809680461884 | | validate/pred_loss_epoch | 0.008976809680461884 | | validate/sigreg_loss | 1.5434002876281738 | | validate/sigreg_loss_epoch | 1.5434002876281738 | +----------------------------+------------------------+ 14:06:31 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_003.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 14:06:31 | INFO | checkpoint_~| Checking for non PyTorch modules to save 14:06:31 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 14:06:31 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.27s write=0.13s rename=0.9ms) [Epoch 2/10] done in 2315.7s 14:06:31 | INFO | checkpoint_~| Checking for non PyTorch modules to save 14:06:31 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=2-step=41799.ckpt 14:06:32 | INFO | atomic_chec~| [atomic_save] ✓ epoch=2-step=41799.ckpt saved (206.8 MiB, serialise=0.26s write=0.11s rename=0.1ms) 14:06:32 | INFO | checkpoint_~| Checking for non PyTorch modules to save 14:06:32 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 14:06:32 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.26s write=0.14s rename=1.5ms) [Epoch 3/10] step 50/13933 (5.7 it/s) [Epoch 3/10] step 100/13933 (6.3 it/s) [Epoch 3/10] step 150/13933 (6.6 it/s) [Epoch 3/10] step 200/13933 (6.7 it/s) [Epoch 3/10] step 250/13933 (6.8 it/s) [Epoch 3/10] step 300/13933 (6.9 it/s) [Epoch 3/10] step 350/13933 (6.9 it/s) [Epoch 3/10] step 400/13933 (6.9 it/s) [Epoch 3/10] step 450/13933 (7.0 it/s) [Epoch 3/10] step 500/13933 (7.0 it/s) [Epoch 3/10] step 550/13933 (7.0 it/s) [Epoch 3/10] step 600/13933 (7.0 it/s) [Epoch 3/10] step 650/13933 (7.0 it/s) [Epoch 3/10] step 700/13933 (7.0 it/s) [Epoch 3/10] step 750/13933 (7.0 it/s) [Epoch 3/10] step 800/13933 (7.0 it/s) [Epoch 3/10] step 850/13933 (7.0 it/s) [Epoch 3/10] step 900/13933 (7.0 it/s) [Epoch 3/10] step 950/13933 (7.0 it/s) [Epoch 3/10] step 1000/13933 (7.0 it/s) [Epoch 3/10] step 1050/13933 (7.0 it/s) [Epoch 3/10] step 1100/13933 (7.0 it/s) [Epoch 3/10] step 1150/13933 (7.0 it/s) [Epoch 3/10] step 1200/13933 (7.0 it/s) [Epoch 3/10] step 1250/13933 (7.0 it/s) [Epoch 3/10] step 1300/13933 (7.0 it/s) [Epoch 3/10] step 1350/13933 (7.0 it/s) [Epoch 3/10] step 1400/13933 (7.0 it/s) [Epoch 3/10] step 1450/13933 (7.0 it/s) [Epoch 3/10] step 1500/13933 (7.0 it/s) [Epoch 3/10] step 1550/13933 (7.0 it/s) [Epoch 3/10] step 1600/13933 (7.0 it/s) [Epoch 3/10] step 1650/13933 (7.0 it/s) [Epoch 3/10] step 1700/13933 (7.0 it/s) [Epoch 3/10] step 1750/13933 (7.0 it/s) [Epoch 3/10] step 1800/13933 (7.0 it/s) [Epoch 3/10] step 1850/13933 (7.0 it/s) [Epoch 3/10] step 1900/13933 (7.0 it/s) [Epoch 3/10] step 1950/13933 (7.0 it/s) [Epoch 3/10] step 2000/13933 (7.0 it/s) [Epoch 3/10] step 2050/13933 (7.0 it/s) [Epoch 3/10] step 2100/13933 (7.0 it/s) [Epoch 3/10] step 2150/13933 (7.0 it/s) [Epoch 3/10] step 2200/13933 (7.0 it/s) [Epoch 3/10] step 2250/13933 (7.0 it/s) [Epoch 3/10] step 2300/13933 (7.0 it/s) [Epoch 3/10] step 2350/13933 (7.0 it/s) [Epoch 3/10] step 2400/13933 (7.0 it/s) [Epoch 3/10] step 2450/13933 (7.0 it/s) [Epoch 3/10] step 2500/13933 (7.0 it/s) [Epoch 3/10] step 2550/13933 (7.0 it/s) [Epoch 3/10] step 2600/13933 (7.0 it/s) [Epoch 3/10] step 2650/13933 (7.0 it/s) [Epoch 3/10] step 2700/13933 (7.0 it/s) [Epoch 3/10] step 2750/13933 (7.0 it/s) [Epoch 3/10] step 2800/13933 (7.0 it/s) [Epoch 3/10] step 2850/13933 (7.0 it/s) [Epoch 3/10] step 2900/13933 (7.0 it/s) [Epoch 3/10] step 2950/13933 (7.0 it/s) [Epoch 3/10] step 3000/13933 (7.0 it/s) [Epoch 3/10] step 3050/13933 (7.0 it/s) [Epoch 3/10] step 3100/13933 (7.0 it/s) [Epoch 3/10] step 3150/13933 (7.0 it/s) [Epoch 3/10] step 3200/13933 (7.0 it/s) [Epoch 3/10] step 3250/13933 (7.0 it/s) [Epoch 3/10] step 3300/13933 (7.0 it/s) [Epoch 3/10] step 3350/13933 (7.0 it/s) [Epoch 3/10] step 3400/13933 (7.0 it/s) [Epoch 3/10] step 3450/13933 (7.0 it/s) [Epoch 3/10] step 3500/13933 (7.0 it/s) [Epoch 3/10] step 3550/13933 (7.0 it/s) [Epoch 3/10] step 3600/13933 (7.0 it/s) [Epoch 3/10] step 3650/13933 (7.0 it/s) [Epoch 3/10] step 3700/13933 (7.0 it/s) [Epoch 3/10] step 3750/13933 (7.0 it/s) [Epoch 3/10] step 3800/13933 (7.0 it/s) [Epoch 3/10] step 3850/13933 (7.0 it/s) [Epoch 3/10] step 3900/13933 (7.0 it/s) [Epoch 3/10] step 3950/13933 (7.0 it/s) [Epoch 3/10] step 4000/13933 (7.0 it/s) [Epoch 3/10] step 4050/13933 (7.0 it/s) [Epoch 3/10] step 4100/13933 (7.0 it/s) [Epoch 3/10] step 4150/13933 (7.0 it/s) [Epoch 3/10] step 4200/13933 (7.0 it/s) [Epoch 3/10] step 4250/13933 (7.0 it/s) [Epoch 3/10] step 4300/13933 (7.0 it/s) [Epoch 3/10] step 4350/13933 (7.0 it/s) [Epoch 3/10] step 4400/13933 (7.0 it/s) [Epoch 3/10] step 4450/13933 (7.0 it/s) [Epoch 3/10] step 4500/13933 (7.0 it/s) [Epoch 3/10] step 4550/13933 (7.0 it/s) [Epoch 3/10] step 4600/13933 (7.0 it/s) [Epoch 3/10] step 4650/13933 (7.0 it/s) [Epoch 3/10] step 4700/13933 (7.0 it/s) [Epoch 3/10] step 4750/13933 (7.0 it/s) [Epoch 3/10] step 4800/13933 (7.0 it/s) [Epoch 3/10] step 4850/13933 (7.0 it/s) [Epoch 3/10] step 4900/13933 (7.0 it/s) [Epoch 3/10] step 4950/13933 (7.0 it/s) [Epoch 3/10] step 5000/13933 (7.0 it/s) [Epoch 3/10] step 5050/13933 (7.0 it/s) [Epoch 3/10] step 5100/13933 (7.0 it/s) [Epoch 3/10] step 5150/13933 (7.0 it/s) [Epoch 3/10] step 5200/13933 (7.0 it/s) [Epoch 3/10] step 5250/13933 (7.0 it/s) [Epoch 3/10] step 5300/13933 (7.0 it/s) [Epoch 3/10] step 5350/13933 (7.0 it/s) [Epoch 3/10] step 5400/13933 (7.0 it/s) [Epoch 3/10] step 5450/13933 (7.0 it/s) [Epoch 3/10] step 5500/13933 (7.0 it/s) [Epoch 3/10] step 5550/13933 (7.0 it/s) [Epoch 3/10] step 5600/13933 (7.0 it/s) [Epoch 3/10] step 5650/13933 (7.0 it/s) [Epoch 3/10] step 5700/13933 (7.0 it/s) [Epoch 3/10] step 5750/13933 (7.0 it/s) [Epoch 3/10] step 5800/13933 (7.0 it/s) [Epoch 3/10] step 5850/13933 (7.0 it/s) [Epoch 3/10] step 5900/13933 (7.0 it/s) [Epoch 3/10] step 5950/13933 (7.0 it/s) [Epoch 3/10] step 6000/13933 (7.0 it/s) [Epoch 3/10] step 6050/13933 (7.0 it/s) [Epoch 3/10] step 6100/13933 (7.0 it/s) [Epoch 3/10] step 6150/13933 (7.0 it/s) [Epoch 3/10] step 6200/13933 (7.0 it/s) [Epoch 3/10] step 6250/13933 (7.0 it/s) [Epoch 3/10] step 6300/13933 (7.0 it/s) [Epoch 3/10] step 6350/13933 (7.0 it/s) [Epoch 3/10] step 6400/13933 (7.0 it/s) [Epoch 3/10] step 6450/13933 (7.0 it/s) [Epoch 3/10] step 6500/13933 (7.0 it/s) [Epoch 3/10] step 6550/13933 (7.0 it/s) [Epoch 3/10] step 6600/13933 (7.0 it/s) [Epoch 3/10] step 6650/13933 (7.0 it/s) [Epoch 3/10] step 6700/13933 (7.0 it/s) [Epoch 3/10] step 6750/13933 (7.0 it/s) [Epoch 3/10] step 6800/13933 (7.0 it/s) [Epoch 3/10] step 6850/13933 (7.0 it/s) [Epoch 3/10] step 6900/13933 (7.0 it/s) [Epoch 3/10] step 6950/13933 (7.0 it/s) [Epoch 3/10] step 7000/13933 (7.0 it/s) [Epoch 3/10] step 7050/13933 (7.0 it/s) [Epoch 3/10] step 7100/13933 (7.0 it/s) [Epoch 3/10] step 7150/13933 (7.0 it/s) [Epoch 3/10] step 7200/13933 (7.0 it/s) [Epoch 3/10] step 7250/13933 (7.0 it/s) [Epoch 3/10] step 7300/13933 (7.0 it/s) [Epoch 3/10] step 7350/13933 (7.0 it/s) [Epoch 3/10] step 7400/13933 (7.0 it/s) [Epoch 3/10] step 7450/13933 (7.0 it/s) [Epoch 3/10] step 7500/13933 (7.0 it/s) [Epoch 3/10] step 7550/13933 (7.0 it/s) [Epoch 3/10] step 7600/13933 (7.0 it/s) [Epoch 3/10] step 7650/13933 (7.0 it/s) [Epoch 3/10] step 7700/13933 (7.0 it/s) [Epoch 3/10] step 7750/13933 (7.0 it/s) [Epoch 3/10] step 7800/13933 (7.0 it/s) [Epoch 3/10] step 7850/13933 (7.0 it/s) [Epoch 3/10] step 7900/13933 (7.0 it/s) [Epoch 3/10] step 7950/13933 (7.0 it/s) [Epoch 3/10] step 8000/13933 (7.0 it/s) [Epoch 3/10] step 8050/13933 (7.0 it/s) [Epoch 3/10] step 8100/13933 (7.0 it/s) [Epoch 3/10] step 8150/13933 (7.0 it/s) [Epoch 3/10] step 8200/13933 (7.0 it/s) [Epoch 3/10] step 8250/13933 (7.0 it/s) [Epoch 3/10] step 8300/13933 (7.0 it/s) [Epoch 3/10] step 8350/13933 (7.0 it/s) [Epoch 3/10] step 8400/13933 (7.0 it/s) [Epoch 3/10] step 8450/13933 (7.0 it/s) [Epoch 3/10] step 8500/13933 (7.0 it/s) [Epoch 3/10] step 8550/13933 (7.0 it/s) [Epoch 3/10] step 8600/13933 (7.0 it/s) [Epoch 3/10] step 8650/13933 (7.0 it/s) [Epoch 3/10] step 8700/13933 (7.0 it/s) [Epoch 3/10] step 8750/13933 (7.0 it/s) [Epoch 3/10] step 8800/13933 (7.0 it/s) [Epoch 3/10] step 8850/13933 (7.0 it/s) [Epoch 3/10] step 8900/13933 (7.0 it/s) [Epoch 3/10] step 8950/13933 (7.0 it/s) [Epoch 3/10] step 9000/13933 (7.0 it/s) [Epoch 3/10] step 9050/13933 (7.0 it/s) [Epoch 3/10] step 9100/13933 (7.0 it/s) [Epoch 3/10] step 9150/13933 (7.0 it/s) [Epoch 3/10] step 9200/13933 (7.0 it/s) [Epoch 3/10] step 9250/13933 (7.0 it/s) [Epoch 3/10] step 9300/13933 (7.0 it/s) [Epoch 3/10] step 9350/13933 (7.0 it/s) [Epoch 3/10] step 9400/13933 (7.0 it/s) [Epoch 3/10] step 9450/13933 (7.0 it/s) [Epoch 3/10] step 9500/13933 (7.0 it/s) [Epoch 3/10] step 9550/13933 (7.0 it/s) [Epoch 3/10] step 9600/13933 (7.0 it/s) [Epoch 3/10] step 9650/13933 (7.0 it/s) [Epoch 3/10] step 9700/13933 (7.0 it/s) [Epoch 3/10] step 9750/13933 (7.0 it/s) [Epoch 3/10] step 9800/13933 (7.0 it/s) [Epoch 3/10] step 9850/13933 (7.0 it/s) [Epoch 3/10] step 9900/13933 (7.0 it/s) [Epoch 3/10] step 9950/13933 (7.0 it/s) [Epoch 3/10] step 10000/13933 (7.0 it/s) [Epoch 3/10] step 10050/13933 (7.0 it/s) [Epoch 3/10] step 10100/13933 (7.0 it/s) [Epoch 3/10] step 10150/13933 (7.0 it/s) [Epoch 3/10] step 10200/13933 (7.0 it/s) [Epoch 3/10] step 10250/13933 (7.0 it/s) [Epoch 3/10] step 10300/13933 (7.0 it/s) [Epoch 3/10] step 10350/13933 (7.0 it/s) [Epoch 3/10] step 10400/13933 (7.0 it/s) [Epoch 3/10] step 10450/13933 (7.0 it/s) [Epoch 3/10] step 10500/13933 (7.0 it/s) [Epoch 3/10] step 10550/13933 (7.0 it/s) [Epoch 3/10] step 10600/13933 (7.0 it/s) [Epoch 3/10] step 10650/13933 (7.0 it/s) [Epoch 3/10] step 10700/13933 (7.0 it/s) [Epoch 3/10] step 10750/13933 (7.0 it/s) [Epoch 3/10] step 10800/13933 (7.0 it/s) [Epoch 3/10] step 10850/13933 (7.0 it/s) [Epoch 3/10] step 10900/13933 (7.0 it/s) [Epoch 3/10] step 10950/13933 (7.0 it/s) [Epoch 3/10] step 11000/13933 (7.0 it/s) [Epoch 3/10] step 11050/13933 (7.0 it/s) [Epoch 3/10] step 11100/13933 (7.0 it/s) [Epoch 3/10] step 11150/13933 (7.0 it/s) [Epoch 3/10] step 11200/13933 (7.0 it/s) [Epoch 3/10] step 11250/13933 (7.0 it/s) [Epoch 3/10] step 11300/13933 (7.0 it/s) [Epoch 3/10] step 11350/13933 (7.0 it/s) [Epoch 3/10] step 11400/13933 (7.0 it/s) [Epoch 3/10] step 11450/13933 (7.0 it/s) [Epoch 3/10] step 11500/13933 (7.0 it/s) [Epoch 3/10] step 11550/13933 (7.0 it/s) [Epoch 3/10] step 11600/13933 (7.0 it/s) [Epoch 3/10] step 11650/13933 (7.0 it/s) [Epoch 3/10] step 11700/13933 (7.0 it/s) [Epoch 3/10] step 11750/13933 (7.0 it/s) [Epoch 3/10] step 11800/13933 (7.0 it/s) [Epoch 3/10] step 11850/13933 (7.0 it/s) [Epoch 3/10] step 11900/13933 (7.0 it/s) [Epoch 3/10] step 11950/13933 (7.0 it/s) [Epoch 3/10] step 12000/13933 (7.0 it/s) [Epoch 3/10] step 12050/13933 (7.0 it/s) [Epoch 3/10] step 12100/13933 (7.0 it/s) [Epoch 3/10] step 12150/13933 (7.0 it/s) [Epoch 3/10] step 12200/13933 (7.0 it/s) [Epoch 3/10] step 12250/13933 (7.0 it/s) [Epoch 3/10] step 12300/13933 (7.0 it/s) [Epoch 3/10] step 12350/13933 (7.0 it/s) [Epoch 3/10] step 12400/13933 (7.0 it/s) [Epoch 3/10] step 12450/13933 (7.0 it/s) [Epoch 3/10] step 12500/13933 (7.0 it/s) [Epoch 3/10] step 12550/13933 (7.0 it/s) [Epoch 3/10] step 12600/13933 (7.0 it/s) [Epoch 3/10] step 12650/13933 (7.0 it/s) [Epoch 3/10] step 12700/13933 (7.0 it/s) [Epoch 3/10] step 12750/13933 (7.0 it/s) [Epoch 3/10] step 12800/13933 (7.0 it/s) [Epoch 3/10] step 12850/13933 (7.0 it/s) [Epoch 3/10] step 12900/13933 (7.0 it/s) [Epoch 3/10] step 12950/13933 (7.0 it/s) [Epoch 3/10] step 13000/13933 (7.0 it/s) [Epoch 3/10] step 13050/13933 (7.0 it/s) [Epoch 3/10] step 13100/13933 (7.0 it/s) [Epoch 3/10] step 13150/13933 (7.0 it/s) [Epoch 3/10] step 13200/13933 (7.0 it/s) [Epoch 3/10] step 13250/13933 (7.0 it/s) [Epoch 3/10] step 13300/13933 (7.0 it/s) [Epoch 3/10] step 13350/13933 (7.0 it/s) [Epoch 3/10] step 13400/13933 (7.0 it/s) [Epoch 3/10] step 13450/13933 (7.0 it/s) [Epoch 3/10] step 13500/13933 (7.0 it/s) [Epoch 3/10] step 13550/13933 (7.0 it/s) [Epoch 3/10] step 13600/13933 (7.0 it/s) [Epoch 3/10] step 13650/13933 (7.0 it/s) [Epoch 3/10] step 13700/13933 (7.0 it/s) [Epoch 3/10] step 13750/13933 (7.0 it/s) [Epoch 3/10] step 13800/13933 (7.0 it/s) [Epoch 3/10] step 13850/13933 (7.0 it/s) [Epoch 3/10] step 13900/13933 (7.0 it/s) 14:43:36 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.0964086726307869 | | fit/pred_loss | 0.009982890449464321 | | fit/sigreg_loss | 0.9609375 | | hardware/cpu_percent | 20.100000381469727 | | hardware/disk_read_mb_s | 2423.913818359375 | | hardware/disk_write_mb_s | 5.027189254760742 | | hardware/net_recv_mb_s | 0.00015176984015852213 | | hardware/net_sent_mb_s | 0.0002675941796042025 | | hardware/ram_used_gb | 223.97030639648438 | | hardware/ram_used_pct | 44.5 | | hparams/lr_model_opt | 3.317660230095498e-05 | | validate/loss | 0.1352843940258026 | | validate/loss_epoch | 0.1352843940258026 | | validate/pred_loss | 0.006805163808166981 | | validate/pred_loss_epoch | 0.006805163808166981 | | validate/sigreg_loss | 1.4275161027908325 | | validate/sigreg_loss_epoch | 1.4275161027908325 | +----------------------------+------------------------+ 14:43:36 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_004.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 14:43:36 | INFO | checkpoint_~| Checking for non PyTorch modules to save 14:43:36 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 14:43:37 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.28s write=0.16s rename=0.2ms) [Epoch 3/10] done in 2224.7s 14:43:37 | INFO | checkpoint_~| Checking for non PyTorch modules to save 14:43:37 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=3-step=55732.ckpt 14:43:37 | INFO | atomic_chec~| [atomic_save] ✓ epoch=3-step=55732.ckpt saved (206.8 MiB, serialise=0.26s write=0.10s rename=0.1ms) 14:43:37 | INFO | checkpoint_~| Checking for non PyTorch modules to save 14:43:37 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 14:43:37 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.25s write=0.10s rename=1.3ms) [Epoch 4/10] step 50/13933 (5.4 it/s) [Epoch 4/10] step 100/13933 (5.4 it/s) [Epoch 4/10] step 150/13933 (5.3 it/s) [Epoch 4/10] step 200/13933 (5.1 it/s) [Epoch 4/10] step 250/13933 (5.2 it/s) [Epoch 4/10] step 300/13933 (5.4 it/s) [Epoch 4/10] step 350/13933 (5.6 it/s) [Epoch 4/10] step 400/13933 (5.8 it/s) [Epoch 4/10] step 450/13933 (5.9 it/s) [Epoch 4/10] step 500/13933 (6.0 it/s) [Epoch 4/10] step 550/13933 (6.0 it/s) [Epoch 4/10] step 600/13933 (5.9 it/s) [Epoch 4/10] step 650/13933 (5.9 it/s) [Epoch 4/10] step 700/13933 (6.0 it/s) [Epoch 4/10] step 750/13933 (6.1 it/s) [Epoch 4/10] step 800/13933 (6.1 it/s) [Epoch 4/10] step 850/13933 (6.1 it/s) [Epoch 4/10] step 900/13933 (6.1 it/s) [Epoch 4/10] step 950/13933 (6.1 it/s) [Epoch 4/10] step 1000/13933 (6.2 it/s) [Epoch 4/10] step 1050/13933 (6.2 it/s) [Epoch 4/10] step 1100/13933 (6.2 it/s) [Epoch 4/10] step 1150/13933 (6.3 it/s) [Epoch 4/10] step 1200/13933 (6.3 it/s) [Epoch 4/10] step 1250/13933 (6.3 it/s) [Epoch 4/10] step 1300/13933 (6.3 it/s) [Epoch 4/10] step 1350/13933 (6.4 it/s) [Epoch 4/10] step 1400/13933 (6.4 it/s) [Epoch 4/10] step 1450/13933 (6.4 it/s) [Epoch 4/10] step 1500/13933 (6.4 it/s) [Epoch 4/10] step 1550/13933 (6.4 it/s) [Epoch 4/10] step 1600/13933 (6.4 it/s) [Epoch 4/10] step 1650/13933 (6.4 it/s) [Epoch 4/10] step 1700/13933 (6.4 it/s) [Epoch 4/10] step 1750/13933 (6.4 it/s) [Epoch 4/10] step 1800/13933 (6.5 it/s) [Epoch 4/10] step 1850/13933 (6.5 it/s) [Epoch 4/10] step 1900/13933 (6.5 it/s) [Epoch 4/10] step 1950/13933 (6.5 it/s) [Epoch 4/10] step 2000/13933 (6.5 it/s) [Epoch 4/10] step 2050/13933 (6.5 it/s) [Epoch 4/10] step 2100/13933 (6.5 it/s) [Epoch 4/10] step 2150/13933 (6.5 it/s) [Epoch 4/10] step 2200/13933 (6.5 it/s) [Epoch 4/10] step 2250/13933 (6.5 it/s) [Epoch 4/10] step 2300/13933 (6.5 it/s) [Epoch 4/10] step 2350/13933 (6.6 it/s) [Epoch 4/10] step 2400/13933 (6.6 it/s) [Epoch 4/10] step 2450/13933 (6.6 it/s) [Epoch 4/10] step 2500/13933 (6.6 it/s) [Epoch 4/10] step 2550/13933 (6.6 it/s) [Epoch 4/10] step 2600/13933 (6.6 it/s) [Epoch 4/10] step 2650/13933 (6.6 it/s) [Epoch 4/10] step 2700/13933 (6.6 it/s) [Epoch 4/10] step 2750/13933 (6.6 it/s) [Epoch 4/10] step 2800/13933 (6.6 it/s) [Epoch 4/10] step 2850/13933 (6.6 it/s) [Epoch 4/10] step 2900/13933 (6.6 it/s) [Epoch 4/10] step 2950/13933 (6.6 it/s) [Epoch 4/10] step 3000/13933 (6.6 it/s) [Epoch 4/10] step 3050/13933 (6.7 it/s) [Epoch 4/10] step 3100/13933 (6.7 it/s) [Epoch 4/10] step 3150/13933 (6.7 it/s) [Epoch 4/10] step 3200/13933 (6.7 it/s) [Epoch 4/10] step 3250/13933 (6.7 it/s) [Epoch 4/10] step 3300/13933 (6.7 it/s) [Epoch 4/10] step 3350/13933 (6.7 it/s) [Epoch 4/10] step 3400/13933 (6.7 it/s) [Epoch 4/10] step 3450/13933 (6.7 it/s) [Epoch 4/10] step 3500/13933 (6.7 it/s) [Epoch 4/10] step 3550/13933 (6.7 it/s) [Epoch 4/10] step 3600/13933 (6.7 it/s) [Epoch 4/10] step 3650/13933 (6.7 it/s) [Epoch 4/10] step 3700/13933 (6.7 it/s) [Epoch 4/10] step 3750/13933 (6.7 it/s) [Epoch 4/10] step 3800/13933 (6.7 it/s) [Epoch 4/10] step 3850/13933 (6.7 it/s) [Epoch 4/10] step 3900/13933 (6.7 it/s) [Epoch 4/10] step 3950/13933 (6.7 it/s) [Epoch 4/10] step 4000/13933 (6.7 it/s) [Epoch 4/10] step 4050/13933 (6.7 it/s) [Epoch 4/10] step 4100/13933 (6.7 it/s) [Epoch 4/10] step 4150/13933 (6.7 it/s) [Epoch 4/10] step 4200/13933 (6.7 it/s) [Epoch 4/10] step 4250/13933 (6.7 it/s) [Epoch 4/10] step 4300/13933 (6.7 it/s) [Epoch 4/10] step 4350/13933 (6.8 it/s) [Epoch 4/10] step 4400/13933 (6.8 it/s) [Epoch 4/10] step 4450/13933 (6.8 it/s) [Epoch 4/10] step 4500/13933 (6.8 it/s) [Epoch 4/10] step 4550/13933 (6.8 it/s) [Epoch 4/10] step 4600/13933 (6.8 it/s) [Epoch 4/10] step 4650/13933 (6.8 it/s) [Epoch 4/10] step 4700/13933 (6.8 it/s) [Epoch 4/10] step 4750/13933 (6.7 it/s) [Epoch 4/10] step 4800/13933 (6.7 it/s) [Epoch 4/10] step 4850/13933 (6.7 it/s) [Epoch 4/10] step 4900/13933 (6.7 it/s) [Epoch 4/10] step 4950/13933 (6.7 it/s) [Epoch 4/10] step 5000/13933 (6.7 it/s) [Epoch 4/10] step 5050/13933 (6.7 it/s) [Epoch 4/10] step 5100/13933 (6.7 it/s) [Epoch 4/10] step 5150/13933 (6.7 it/s) [Epoch 4/10] step 5200/13933 (6.7 it/s) [Epoch 4/10] step 5250/13933 (6.7 it/s) [Epoch 4/10] step 5300/13933 (6.7 it/s) [Epoch 4/10] step 5350/13933 (6.6 it/s) [Epoch 4/10] step 5400/13933 (6.6 it/s) [Epoch 4/10] step 5450/13933 (6.6 it/s) [Epoch 4/10] step 5500/13933 (6.6 it/s) [Epoch 4/10] step 5550/13933 (6.6 it/s) [Epoch 4/10] step 5600/13933 (6.6 it/s) [Epoch 4/10] step 5650/13933 (6.6 it/s) [Epoch 4/10] step 5700/13933 (6.6 it/s) [Epoch 4/10] step 5750/13933 (6.6 it/s) [Epoch 4/10] step 5800/13933 (6.6 it/s) [Epoch 4/10] step 5850/13933 (6.6 it/s) [Epoch 4/10] step 5900/13933 (6.6 it/s) [Epoch 4/10] step 5950/13933 (6.6 it/s) [Epoch 4/10] step 6000/13933 (6.6 it/s) [Epoch 4/10] step 6050/13933 (6.6 it/s) [Epoch 4/10] step 6100/13933 (6.6 it/s) [Epoch 4/10] step 6150/13933 (6.6 it/s) [Epoch 4/10] step 6200/13933 (6.6 it/s) [Epoch 4/10] step 6250/13933 (6.6 it/s) [Epoch 4/10] step 6300/13933 (6.6 it/s) [Epoch 4/10] step 6350/13933 (6.6 it/s) [Epoch 4/10] step 6400/13933 (6.6 it/s) [Epoch 4/10] step 6450/13933 (6.6 it/s) [Epoch 4/10] step 6500/13933 (6.6 it/s) [Epoch 4/10] step 6550/13933 (6.6 it/s) [Epoch 4/10] step 6600/13933 (6.6 it/s) [Epoch 4/10] step 6650/13933 (6.6 it/s) [Epoch 4/10] step 6700/13933 (6.6 it/s) [Epoch 4/10] step 6750/13933 (6.6 it/s) [Epoch 4/10] step 6800/13933 (6.6 it/s) [Epoch 4/10] step 6850/13933 (6.6 it/s) [Epoch 4/10] step 6900/13933 (6.6 it/s) [Epoch 4/10] step 6950/13933 (6.6 it/s) [Epoch 4/10] step 7000/13933 (6.6 it/s) [Epoch 4/10] step 7050/13933 (6.6 it/s) [Epoch 4/10] step 7100/13933 (6.6 it/s) [Epoch 4/10] step 7150/13933 (6.6 it/s) [Epoch 4/10] step 7200/13933 (6.6 it/s) [Epoch 4/10] step 7250/13933 (6.6 it/s) [Epoch 4/10] step 7300/13933 (6.6 it/s) [Epoch 4/10] step 7350/13933 (6.6 it/s) [Epoch 4/10] step 7400/13933 (6.6 it/s) [Epoch 4/10] step 7450/13933 (6.6 it/s) [Epoch 4/10] step 7500/13933 (6.6 it/s) [Epoch 4/10] step 7550/13933 (6.6 it/s) [Epoch 4/10] step 7600/13933 (6.6 it/s) [Epoch 4/10] step 7650/13933 (6.6 it/s) [Epoch 4/10] step 7700/13933 (6.6 it/s) [Epoch 4/10] step 7750/13933 (6.6 it/s) [Epoch 4/10] step 7800/13933 (6.6 it/s) [Epoch 4/10] step 7850/13933 (6.6 it/s) [Epoch 4/10] step 7900/13933 (6.6 it/s) [Epoch 4/10] step 7950/13933 (6.6 it/s) [Epoch 4/10] step 8000/13933 (6.6 it/s) [Epoch 4/10] step 8050/13933 (6.6 it/s) [Epoch 4/10] step 8100/13933 (6.6 it/s) [Epoch 4/10] step 8150/13933 (6.6 it/s) [Epoch 4/10] step 8200/13933 (6.6 it/s) [Epoch 4/10] step 8250/13933 (6.6 it/s) [Epoch 4/10] step 8300/13933 (6.6 it/s) [Epoch 4/10] step 8350/13933 (6.6 it/s) [Epoch 4/10] step 8400/13933 (6.6 it/s) [Epoch 4/10] step 8450/13933 (6.6 it/s) [Epoch 4/10] step 8500/13933 (6.6 it/s) [Epoch 4/10] step 8550/13933 (6.5 it/s) [Epoch 4/10] step 8600/13933 (6.5 it/s) [Epoch 4/10] step 8650/13933 (6.5 it/s) [Epoch 4/10] step 8700/13933 (6.5 it/s) [Epoch 4/10] step 8750/13933 (6.5 it/s) [Epoch 4/10] step 8800/13933 (6.5 it/s) [Epoch 4/10] step 8850/13933 (6.5 it/s) [Epoch 4/10] step 8900/13933 (6.5 it/s) [Epoch 4/10] step 8950/13933 (6.5 it/s) [Epoch 4/10] step 9000/13933 (6.5 it/s) [Epoch 4/10] step 9050/13933 (6.5 it/s) [Epoch 4/10] step 9100/13933 (6.5 it/s) [Epoch 4/10] step 9150/13933 (6.5 it/s) [Epoch 4/10] step 9200/13933 (6.5 it/s) [Epoch 4/10] step 9250/13933 (6.5 it/s) [Epoch 4/10] step 9300/13933 (6.5 it/s) [Epoch 4/10] step 9350/13933 (6.5 it/s) [Epoch 4/10] step 9400/13933 (6.5 it/s) [Epoch 4/10] step 9450/13933 (6.5 it/s) [Epoch 4/10] step 9500/13933 (6.5 it/s) [Epoch 4/10] step 9550/13933 (6.5 it/s) [Epoch 4/10] step 9600/13933 (6.5 it/s) [Epoch 4/10] step 9650/13933 (6.5 it/s) [Epoch 4/10] step 9700/13933 (6.5 it/s) [Epoch 4/10] step 9750/13933 (6.5 it/s) [Epoch 4/10] step 9800/13933 (6.5 it/s) [Epoch 4/10] step 9850/13933 (6.5 it/s) [Epoch 4/10] step 9900/13933 (6.5 it/s) [Epoch 4/10] step 9950/13933 (6.5 it/s) [Epoch 4/10] step 10000/13933 (6.5 it/s) [Epoch 4/10] step 10050/13933 (6.5 it/s) [Epoch 4/10] step 10100/13933 (6.5 it/s) [Epoch 4/10] step 10150/13933 (6.5 it/s) [Epoch 4/10] step 10200/13933 (6.5 it/s) [Epoch 4/10] step 10250/13933 (6.5 it/s) [Epoch 4/10] step 10300/13933 (6.5 it/s) [Epoch 4/10] step 10350/13933 (6.5 it/s) [Epoch 4/10] step 10400/13933 (6.5 it/s) [Epoch 4/10] step 10450/13933 (6.5 it/s) [Epoch 4/10] step 10500/13933 (6.5 it/s) [Epoch 4/10] step 10550/13933 (6.5 it/s) [Epoch 4/10] step 10600/13933 (6.5 it/s) [Epoch 4/10] step 10650/13933 (6.5 it/s) [Epoch 4/10] step 10700/13933 (6.5 it/s) [Epoch 4/10] step 10750/13933 (6.5 it/s) [Epoch 4/10] step 10800/13933 (6.5 it/s) [Epoch 4/10] step 10850/13933 (6.5 it/s) [Epoch 4/10] step 10900/13933 (6.5 it/s) [Epoch 4/10] step 10950/13933 (6.5 it/s) [Epoch 4/10] step 11000/13933 (6.5 it/s) [Epoch 4/10] step 11050/13933 (6.5 it/s) [Epoch 4/10] step 11100/13933 (6.5 it/s) [Epoch 4/10] step 11150/13933 (6.5 it/s) [Epoch 4/10] step 11200/13933 (6.5 it/s) [Epoch 4/10] step 11250/13933 (6.5 it/s) [Epoch 4/10] step 11300/13933 (6.5 it/s) [Epoch 4/10] step 11350/13933 (6.5 it/s) [Epoch 4/10] step 11400/13933 (6.5 it/s) [Epoch 4/10] step 11450/13933 (6.5 it/s) [Epoch 4/10] step 11500/13933 (6.5 it/s) [Epoch 4/10] step 11550/13933 (6.5 it/s) [Epoch 4/10] step 11600/13933 (6.5 it/s) [Epoch 4/10] step 11650/13933 (6.5 it/s) [Epoch 4/10] step 11700/13933 (6.5 it/s) [Epoch 4/10] step 11750/13933 (6.5 it/s) [Epoch 4/10] step 11800/13933 (6.5 it/s) [Epoch 4/10] step 11850/13933 (6.5 it/s) [Epoch 4/10] step 11900/13933 (6.5 it/s) [Epoch 4/10] step 11950/13933 (6.5 it/s) [Epoch 4/10] step 12000/13933 (6.5 it/s) [Epoch 4/10] step 12050/13933 (6.5 it/s) [Epoch 4/10] step 12100/13933 (6.5 it/s) [Epoch 4/10] step 12150/13933 (6.5 it/s) [Epoch 4/10] step 12200/13933 (6.5 it/s) [Epoch 4/10] step 12250/13933 (6.5 it/s) [Epoch 4/10] step 12300/13933 (6.5 it/s) [Epoch 4/10] step 12350/13933 (6.5 it/s) [Epoch 4/10] step 12400/13933 (6.5 it/s) [Epoch 4/10] step 12450/13933 (6.5 it/s) [Epoch 4/10] step 12500/13933 (6.5 it/s) [Epoch 4/10] step 12550/13933 (6.5 it/s) [Epoch 4/10] step 12600/13933 (6.5 it/s) [Epoch 4/10] step 12650/13933 (6.5 it/s) [Epoch 4/10] step 12700/13933 (6.5 it/s) [Epoch 4/10] step 12750/13933 (6.5 it/s) [Epoch 4/10] step 12800/13933 (6.5 it/s) [Epoch 4/10] step 12850/13933 (6.5 it/s) [Epoch 4/10] step 12900/13933 (6.5 it/s) [Epoch 4/10] step 12950/13933 (6.5 it/s) [Epoch 4/10] step 13000/13933 (6.5 it/s) [Epoch 4/10] step 13050/13933 (6.5 it/s) [Epoch 4/10] step 13100/13933 (6.5 it/s) [Epoch 4/10] step 13150/13933 (6.5 it/s) [Epoch 4/10] step 13200/13933 (6.5 it/s) [Epoch 4/10] step 13250/13933 (6.5 it/s) [Epoch 4/10] step 13300/13933 (6.5 it/s) [Epoch 4/10] step 13350/13933 (6.5 it/s) [Epoch 4/10] step 13400/13933 (6.5 it/s) [Epoch 4/10] step 13450/13933 (6.5 it/s) [Epoch 4/10] step 13500/13933 (6.5 it/s) [Epoch 4/10] step 13550/13933 (6.5 it/s) [Epoch 4/10] step 13600/13933 (6.5 it/s) [Epoch 4/10] step 13650/13933 (6.5 it/s) [Epoch 4/10] step 13700/13933 (6.5 it/s) [Epoch 4/10] step 13750/13933 (6.5 it/s) [Epoch 4/10] step 13800/13933 (6.5 it/s) [Epoch 4/10] step 13850/13933 (6.5 it/s) [Epoch 4/10] step 13900/13933 (6.5 it/s) 15:23:31 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.09310823678970337 | | fit/pred_loss | 0.008147303014993668 | | fit/sigreg_loss | 0.94140625 | | hardware/cpu_percent | 17.399999618530273 | | hardware/disk_read_mb_s | 1690.4527587890625 | | hardware/disk_write_mb_s | 0.18747982382774353 | | hardware/net_recv_mb_s | 0.00017562547873239964 | | hardware/net_sent_mb_s | 0.00029004813404753804 | | hardware/ram_used_gb | 192.06288146972656 | | hardware/ram_used_pct | 38.099998474121094 | | hparams/lr_model_opt | 2.5396562705282122e-05 | | validate/loss | 0.1303527057170868 | | validate/loss_epoch | 0.1303527057170868 | | validate/pred_loss | 0.005473831668496132 | | validate/pred_loss_epoch | 0.005473831668496132 | | validate/sigreg_loss | 1.3875468969345093 | | validate/sigreg_loss_epoch | 1.3875468969345093 | +----------------------------+------------------------+ 15:23:31 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_005.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 15:23:31 | INFO | checkpoint_~| Checking for non PyTorch modules to save 15:23:31 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 15:23:31 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.30s write=0.17s rename=0.2ms) [Epoch 4/10] done in 2394.0s 15:23:32 | INFO | checkpoint_~| Checking for non PyTorch modules to save 15:23:32 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=4-step=69665.ckpt 15:23:32 | INFO | atomic_chec~| [atomic_save] ✓ epoch=4-step=69665.ckpt saved (206.8 MiB, serialise=0.26s write=0.14s rename=0.1ms) 15:23:32 | INFO | checkpoint_~| Checking for non PyTorch modules to save 15:23:32 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 15:23:32 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.33s write=0.12s rename=1.5ms) [Epoch 5/10] step 50/13933 (4.7 it/s) [Epoch 5/10] step 100/13933 (4.9 it/s) [Epoch 5/10] step 150/13933 (5.3 it/s) [Epoch 5/10] step 200/13933 (5.7 it/s) [Epoch 5/10] step 250/13933 (5.9 it/s) [Epoch 5/10] step 300/13933 (6.0 it/s) [Epoch 5/10] step 350/13933 (6.1 it/s) [Epoch 5/10] step 400/13933 (5.9 it/s) [Epoch 5/10] step 450/13933 (5.9 it/s) [Epoch 5/10] step 500/13933 (5.8 it/s) [Epoch 5/10] step 550/13933 (5.9 it/s) [Epoch 5/10] step 600/13933 (6.0 it/s) [Epoch 5/10] step 650/13933 (6.0 it/s) [Epoch 5/10] step 700/13933 (6.0 it/s) [Epoch 5/10] step 750/13933 (6.0 it/s) [Epoch 5/10] step 800/13933 (6.0 it/s) [Epoch 5/10] step 850/13933 (6.0 it/s) [Epoch 5/10] step 900/13933 (6.1 it/s) [Epoch 5/10] step 950/13933 (6.1 it/s) [Epoch 5/10] step 1000/13933 (6.2 it/s) [Epoch 5/10] step 1050/13933 (6.2 it/s) [Epoch 5/10] step 1100/13933 (6.2 it/s) [Epoch 5/10] step 1150/13933 (6.2 it/s) [Epoch 5/10] step 1200/13933 (6.2 it/s) [Epoch 5/10] step 1250/13933 (6.2 it/s) [Epoch 5/10] step 1300/13933 (6.2 it/s) [Epoch 5/10] step 1350/13933 (6.2 it/s) [Epoch 5/10] step 1400/13933 (6.2 it/s) [Epoch 5/10] step 1450/13933 (6.2 it/s) [Epoch 5/10] step 1500/13933 (6.2 it/s) [Epoch 5/10] step 1550/13933 (6.2 it/s) [Epoch 5/10] step 1600/13933 (6.3 it/s) [Epoch 5/10] step 1650/13933 (6.3 it/s) [Epoch 5/10] step 1700/13933 (6.3 it/s) [Epoch 5/10] step 1750/13933 (6.2 it/s) [Epoch 5/10] step 1800/13933 (6.3 it/s) [Epoch 5/10] step 1850/13933 (6.3 it/s) [Epoch 5/10] step 1900/13933 (6.3 it/s) [Epoch 5/10] step 1950/13933 (6.3 it/s) [Epoch 5/10] step 2000/13933 (6.3 it/s) [Epoch 5/10] step 2050/13933 (6.3 it/s) [Epoch 5/10] step 2100/13933 (6.3 it/s) [Epoch 5/10] step 2150/13933 (6.3 it/s) [Epoch 5/10] step 2200/13933 (6.3 it/s) [Epoch 5/10] step 2250/13933 (6.3 it/s) [Epoch 5/10] step 2300/13933 (6.3 it/s) [Epoch 5/10] step 2350/13933 (6.3 it/s) [Epoch 5/10] step 2400/13933 (6.3 it/s) [Epoch 5/10] step 2450/13933 (6.3 it/s) [Epoch 5/10] step 2500/13933 (6.3 it/s) [Epoch 5/10] step 2550/13933 (6.4 it/s) [Epoch 5/10] step 2600/13933 (6.4 it/s) [Epoch 5/10] step 2650/13933 (6.4 it/s) [Epoch 5/10] step 2700/13933 (6.3 it/s) [Epoch 5/10] step 2750/13933 (6.3 it/s) [Epoch 5/10] step 2800/13933 (6.3 it/s) [Epoch 5/10] step 2850/13933 (6.3 it/s) [Epoch 5/10] step 2900/13933 (6.3 it/s) [Epoch 5/10] step 2950/13933 (6.3 it/s) [Epoch 5/10] step 3000/13933 (6.3 it/s) [Epoch 5/10] step 3050/13933 (6.3 it/s) [Epoch 5/10] step 3100/13933 (6.3 it/s) [Epoch 5/10] step 3150/13933 (6.3 it/s) [Epoch 5/10] step 3200/13933 (6.3 it/s) [Epoch 5/10] step 3250/13933 (6.3 it/s) [Epoch 5/10] step 3300/13933 (6.3 it/s) [Epoch 5/10] step 3350/13933 (6.3 it/s) [Epoch 5/10] step 3400/13933 (6.3 it/s) [Epoch 5/10] step 3450/13933 (6.3 it/s) [Epoch 5/10] step 3500/13933 (6.3 it/s) [Epoch 5/10] step 3550/13933 (6.3 it/s) [Epoch 5/10] step 3600/13933 (6.3 it/s) [Epoch 5/10] step 3650/13933 (6.3 it/s) [Epoch 5/10] step 3700/13933 (6.3 it/s) [Epoch 5/10] step 3750/13933 (6.3 it/s) [Epoch 5/10] step 3800/13933 (6.4 it/s) [Epoch 5/10] step 3850/13933 (6.3 it/s) [Epoch 5/10] step 3900/13933 (6.3 it/s) [Epoch 5/10] step 3950/13933 (6.3 it/s) [Epoch 5/10] step 4000/13933 (6.4 it/s) [Epoch 5/10] step 4050/13933 (6.4 it/s) [Epoch 5/10] step 4100/13933 (6.4 it/s) [Epoch 5/10] step 4150/13933 (6.4 it/s) [Epoch 5/10] step 4200/13933 (6.4 it/s) [Epoch 5/10] step 4250/13933 (6.4 it/s) [Epoch 5/10] step 4300/13933 (6.4 it/s) [Epoch 5/10] step 4350/13933 (6.4 it/s) [Epoch 5/10] step 4400/13933 (6.4 it/s) [Epoch 5/10] step 4450/13933 (6.4 it/s) [Epoch 5/10] step 4500/13933 (6.4 it/s) [Epoch 5/10] step 4550/13933 (6.4 it/s) [Epoch 5/10] step 4600/13933 (6.4 it/s) [Epoch 5/10] step 4650/13933 (6.4 it/s) [Epoch 5/10] step 4700/13933 (6.4 it/s) [Epoch 5/10] step 4750/13933 (6.4 it/s) [Epoch 5/10] step 4800/13933 (6.4 it/s) [Epoch 5/10] step 4850/13933 (6.4 it/s) [Epoch 5/10] step 4900/13933 (6.4 it/s) [Epoch 5/10] step 4950/13933 (6.4 it/s) [Epoch 5/10] step 5000/13933 (6.4 it/s) [Epoch 5/10] step 5050/13933 (6.4 it/s) [Epoch 5/10] step 5100/13933 (6.4 it/s) [Epoch 5/10] step 5150/13933 (6.4 it/s) [Epoch 5/10] step 5200/13933 (6.4 it/s) [Epoch 5/10] step 5250/13933 (6.4 it/s) [Epoch 5/10] step 5300/13933 (6.4 it/s) [Epoch 5/10] step 5350/13933 (6.4 it/s) [Epoch 5/10] step 5400/13933 (6.4 it/s) [Epoch 5/10] step 5450/13933 (6.4 it/s) [Epoch 5/10] step 5500/13933 (6.4 it/s) [Epoch 5/10] step 5550/13933 (6.4 it/s) [Epoch 5/10] step 5600/13933 (6.4 it/s) [Epoch 5/10] step 5650/13933 (6.4 it/s) [Epoch 5/10] step 5700/13933 (6.4 it/s) [Epoch 5/10] step 5750/13933 (6.4 it/s) [Epoch 5/10] step 5800/13933 (6.4 it/s) [Epoch 5/10] step 5850/13933 (6.4 it/s) [Epoch 5/10] step 5900/13933 (6.4 it/s) [Epoch 5/10] step 5950/13933 (6.4 it/s) [Epoch 5/10] step 6000/13933 (6.4 it/s) [Epoch 5/10] step 6050/13933 (6.4 it/s) [Epoch 5/10] step 6100/13933 (6.4 it/s) [Epoch 5/10] step 6150/13933 (6.4 it/s) [Epoch 5/10] step 6200/13933 (6.4 it/s) [Epoch 5/10] step 6250/13933 (6.4 it/s) [Epoch 5/10] step 6300/13933 (6.4 it/s) [Epoch 5/10] step 6350/13933 (6.4 it/s) [Epoch 5/10] step 6400/13933 (6.4 it/s) [Epoch 5/10] step 6450/13933 (6.4 it/s) [Epoch 5/10] step 6500/13933 (6.4 it/s) [Epoch 5/10] step 6550/13933 (6.4 it/s) [Epoch 5/10] step 6600/13933 (6.4 it/s) [Epoch 5/10] step 6650/13933 (6.4 it/s) [Epoch 5/10] step 6700/13933 (6.4 it/s) [Epoch 5/10] step 6750/13933 (6.4 it/s) [Epoch 5/10] step 6800/13933 (6.4 it/s) [Epoch 5/10] step 6850/13933 (6.4 it/s) [Epoch 5/10] step 6900/13933 (6.4 it/s) [Epoch 5/10] step 6950/13933 (6.4 it/s) [Epoch 5/10] step 7000/13933 (6.4 it/s) [Epoch 5/10] step 7050/13933 (6.4 it/s) [Epoch 5/10] step 7100/13933 (6.4 it/s) [Epoch 5/10] step 7150/13933 (6.4 it/s) [Epoch 5/10] step 7200/13933 (6.4 it/s) [Epoch 5/10] step 7250/13933 (6.4 it/s) [Epoch 5/10] step 7300/13933 (6.4 it/s) [Epoch 5/10] step 7350/13933 (6.4 it/s) [Epoch 5/10] step 7400/13933 (6.4 it/s) [Epoch 5/10] step 7450/13933 (6.4 it/s) [Epoch 5/10] step 7500/13933 (6.5 it/s) [Epoch 5/10] step 7550/13933 (6.5 it/s) [Epoch 5/10] step 7600/13933 (6.4 it/s) [Epoch 5/10] step 7650/13933 (6.4 it/s) [Epoch 5/10] step 7700/13933 (6.5 it/s) [Epoch 5/10] step 7750/13933 (6.5 it/s) [Epoch 5/10] step 7800/13933 (6.5 it/s) [Epoch 5/10] step 7850/13933 (6.5 it/s) [Epoch 5/10] step 7900/13933 (6.5 it/s) [Epoch 5/10] step 7950/13933 (6.5 it/s) [Epoch 5/10] step 8000/13933 (6.5 it/s) [Epoch 5/10] step 8050/13933 (6.5 it/s) [Epoch 5/10] step 8100/13933 (6.5 it/s) [Epoch 5/10] step 8150/13933 (6.5 it/s) [Epoch 5/10] step 8200/13933 (6.5 it/s) [Epoch 5/10] step 8250/13933 (6.5 it/s) [Epoch 5/10] step 8300/13933 (6.5 it/s) [Epoch 5/10] step 8350/13933 (6.5 it/s) [Epoch 5/10] step 8400/13933 (6.5 it/s) [Epoch 5/10] step 8450/13933 (6.5 it/s) [Epoch 5/10] step 8500/13933 (6.5 it/s) [Epoch 5/10] step 8550/13933 (6.5 it/s) [Epoch 5/10] step 8600/13933 (6.5 it/s) [Epoch 5/10] step 8650/13933 (6.5 it/s) [Epoch 5/10] step 8700/13933 (6.5 it/s) [Epoch 5/10] step 8750/13933 (6.5 it/s) [Epoch 5/10] step 8800/13933 (6.5 it/s) [Epoch 5/10] step 8850/13933 (6.5 it/s) [Epoch 5/10] step 8900/13933 (6.5 it/s) [Epoch 5/10] step 8950/13933 (6.5 it/s) [Epoch 5/10] step 9000/13933 (6.5 it/s) [Epoch 5/10] step 9050/13933 (6.5 it/s) [Epoch 5/10] step 9100/13933 (6.5 it/s) [Epoch 5/10] step 9150/13933 (6.5 it/s) [Epoch 5/10] step 9200/13933 (6.5 it/s) [Epoch 5/10] step 9250/13933 (6.5 it/s) [Epoch 5/10] step 9300/13933 (6.5 it/s) [Epoch 5/10] step 9350/13933 (6.5 it/s) [Epoch 5/10] step 9400/13933 (6.5 it/s) [Epoch 5/10] step 9450/13933 (6.5 it/s) [Epoch 5/10] step 9500/13933 (6.5 it/s) [Epoch 5/10] step 9550/13933 (6.5 it/s) [Epoch 5/10] step 9600/13933 (6.5 it/s) [Epoch 5/10] step 9650/13933 (6.5 it/s) [Epoch 5/10] step 9700/13933 (6.5 it/s) [Epoch 5/10] step 9750/13933 (6.5 it/s) [Epoch 5/10] step 9800/13933 (6.5 it/s) [Epoch 5/10] step 9850/13933 (6.5 it/s) [Epoch 5/10] step 9900/13933 (6.5 it/s) [Epoch 5/10] step 9950/13933 (6.5 it/s) [Epoch 5/10] step 10000/13933 (6.5 it/s) [Epoch 5/10] step 10050/13933 (6.5 it/s) [Epoch 5/10] step 10100/13933 (6.5 it/s) [Epoch 5/10] step 10150/13933 (6.5 it/s) [Epoch 5/10] step 10200/13933 (6.5 it/s) [Epoch 5/10] step 10250/13933 (6.5 it/s) [Epoch 5/10] step 10300/13933 (6.5 it/s) [Epoch 5/10] step 10350/13933 (6.5 it/s) [Epoch 5/10] step 10400/13933 (6.5 it/s) [Epoch 5/10] step 10450/13933 (6.5 it/s) [Epoch 5/10] step 10500/13933 (6.5 it/s) [Epoch 5/10] step 10550/13933 (6.5 it/s) [Epoch 5/10] step 10600/13933 (6.5 it/s) [Epoch 5/10] step 10650/13933 (6.5 it/s) [Epoch 5/10] step 10700/13933 (6.5 it/s) [Epoch 5/10] step 10750/13933 (6.5 it/s) [Epoch 5/10] step 10800/13933 (6.5 it/s) [Epoch 5/10] step 10850/13933 (6.5 it/s) [Epoch 5/10] step 10900/13933 (6.5 it/s) [Epoch 5/10] step 10950/13933 (6.5 it/s) [Epoch 5/10] step 11000/13933 (6.5 it/s) [Epoch 5/10] step 11050/13933 (6.5 it/s) [Epoch 5/10] step 11100/13933 (6.5 it/s) [Epoch 5/10] step 11150/13933 (6.5 it/s) [Epoch 5/10] step 11200/13933 (6.5 it/s) [Epoch 5/10] step 11250/13933 (6.5 it/s) [Epoch 5/10] step 11300/13933 (6.5 it/s) [Epoch 5/10] step 11350/13933 (6.5 it/s) [Epoch 5/10] step 11400/13933 (6.5 it/s) [Epoch 5/10] step 11450/13933 (6.5 it/s) [Epoch 5/10] step 11500/13933 (6.5 it/s) [Epoch 5/10] step 11550/13933 (6.5 it/s) [Epoch 5/10] step 11600/13933 (6.5 it/s) [Epoch 5/10] step 11650/13933 (6.5 it/s) [Epoch 5/10] step 11700/13933 (6.5 it/s) [Epoch 5/10] step 11750/13933 (6.5 it/s) [Epoch 5/10] step 11800/13933 (6.5 it/s) [Epoch 5/10] step 11850/13933 (6.5 it/s) [Epoch 5/10] step 11900/13933 (6.5 it/s) [Epoch 5/10] step 11950/13933 (6.5 it/s) [Epoch 5/10] step 12000/13933 (6.5 it/s) [Epoch 5/10] step 12050/13933 (6.5 it/s) [Epoch 5/10] step 12100/13933 (6.5 it/s) [Epoch 5/10] step 12150/13933 (6.5 it/s) [Epoch 5/10] step 12200/13933 (6.5 it/s) [Epoch 5/10] step 12250/13933 (6.5 it/s) [Epoch 5/10] step 12300/13933 (6.5 it/s) [Epoch 5/10] step 12350/13933 (6.5 it/s) [Epoch 5/10] step 12400/13933 (6.5 it/s) [Epoch 5/10] step 12450/13933 (6.5 it/s) [Epoch 5/10] step 12500/13933 (6.5 it/s) [Epoch 5/10] step 12550/13933 (6.5 it/s) [Epoch 5/10] step 12600/13933 (6.5 it/s) [Epoch 5/10] step 12650/13933 (6.5 it/s) [Epoch 5/10] step 12700/13933 (6.5 it/s) [Epoch 5/10] step 12750/13933 (6.5 it/s) [Epoch 5/10] step 12800/13933 (6.5 it/s) [Epoch 5/10] step 12850/13933 (6.5 it/s) [Epoch 5/10] step 12900/13933 (6.5 it/s) [Epoch 5/10] step 12950/13933 (6.5 it/s) [Epoch 5/10] step 13000/13933 (6.5 it/s) [Epoch 5/10] step 13050/13933 (6.5 it/s) [Epoch 5/10] step 13100/13933 (6.5 it/s) [Epoch 5/10] step 13150/13933 (6.5 it/s) [Epoch 5/10] step 13200/13933 (6.5 it/s) [Epoch 5/10] step 13250/13933 (6.5 it/s) [Epoch 5/10] step 13300/13933 (6.5 it/s) [Epoch 5/10] step 13350/13933 (6.5 it/s) [Epoch 5/10] step 13400/13933 (6.5 it/s) [Epoch 5/10] step 13450/13933 (6.5 it/s) [Epoch 5/10] step 13500/13933 (6.5 it/s) [Epoch 5/10] step 13550/13933 (6.5 it/s) [Epoch 5/10] step 13600/13933 (6.5 it/s) [Epoch 5/10] step 13650/13933 (6.5 it/s) [Epoch 5/10] step 13700/13933 (6.5 it/s) [Epoch 5/10] step 13750/13933 (6.5 it/s) [Epoch 5/10] step 13800/13933 (6.5 it/s) [Epoch 5/10] step 13850/13933 (6.5 it/s) [Epoch 5/10] step 13900/13933 (6.5 it/s) 16:03:10 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.08866970986127853 | | fit/pred_loss | 0.00859158206731081 | | fit/sigreg_loss | 0.890625 | | hardware/cpu_percent | 19.5 | | hardware/disk_read_mb_s | 2088.014404296875 | | hardware/disk_write_mb_s | 2.498476266860962 | | hardware/net_recv_mb_s | 0.0001359185262117535 | | hardware/net_sent_mb_s | 0.00024195019796025008 | | hardware/ram_used_gb | 212.07537841796875 | | hardware/ram_used_pct | 42.099998474121094 | | hparams/lr_model_opt | 1.757692552928347e-05 | | validate/loss | 0.12781526148319244 | | validate/loss_epoch | 0.12781526148319244 | | validate/pred_loss | 0.004796930123120546 | | validate/pred_loss_epoch | 0.004796930123120546 | | validate/sigreg_loss | 1.3669145107269287 | | validate/sigreg_loss_epoch | 1.3669145107269287 | +----------------------------+------------------------+ 16:03:10 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_006.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 16:03:10 | INFO | checkpoint_~| Checking for non PyTorch modules to save 16:03:10 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 16:03:11 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.31s write=0.11s rename=0.1ms) [Epoch 5/10] done in 2378.3s 16:03:11 | INFO | checkpoint_~| Checking for non PyTorch modules to save 16:03:11 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=5-step=83598.ckpt 16:03:11 | INFO | atomic_chec~| [atomic_save] ✓ epoch=5-step=83598.ckpt saved (206.8 MiB, serialise=0.32s write=0.12s rename=0.1ms) 16:03:11 | INFO | checkpoint_~| Checking for non PyTorch modules to save 16:03:11 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 16:03:12 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.33s write=0.12s rename=1.2ms) [Epoch 6/10] step 50/13933 (5.8 it/s) [Epoch 6/10] step 100/13933 (6.3 it/s) [Epoch 6/10] step 150/13933 (6.3 it/s) [Epoch 6/10] step 200/13933 (6.1 it/s) [Epoch 6/10] step 250/13933 (6.0 it/s) [Epoch 6/10] step 300/13933 (6.1 it/s) [Epoch 6/10] step 350/13933 (6.3 it/s) [Epoch 6/10] step 400/13933 (6.3 it/s) [Epoch 6/10] step 450/13933 (6.4 it/s) [Epoch 6/10] step 500/13933 (6.3 it/s) [Epoch 6/10] step 550/13933 (6.2 it/s) [Epoch 6/10] step 600/13933 (6.3 it/s) [Epoch 6/10] step 650/13933 (6.3 it/s) [Epoch 6/10] step 700/13933 (6.4 it/s) [Epoch 6/10] step 750/13933 (6.4 it/s) [Epoch 6/10] step 800/13933 (6.4 it/s) [Epoch 6/10] step 850/13933 (6.3 it/s) [Epoch 6/10] step 900/13933 (6.3 it/s) [Epoch 6/10] step 950/13933 (6.3 it/s) [Epoch 6/10] step 1000/13933 (6.3 it/s) [Epoch 6/10] step 1050/13933 (6.4 it/s) [Epoch 6/10] step 1100/13933 (6.4 it/s) [Epoch 6/10] step 1150/13933 (6.4 it/s) [Epoch 6/10] step 1200/13933 (6.3 it/s) [Epoch 6/10] step 1250/13933 (6.3 it/s) [Epoch 6/10] step 1300/13933 (6.3 it/s) [Epoch 6/10] step 1350/13933 (6.4 it/s) [Epoch 6/10] step 1400/13933 (6.4 it/s) [Epoch 6/10] step 1450/13933 (6.4 it/s) [Epoch 6/10] step 1500/13933 (6.3 it/s) [Epoch 6/10] step 1550/13933 (6.3 it/s) [Epoch 6/10] step 1600/13933 (6.4 it/s) [Epoch 6/10] step 1650/13933 (6.4 it/s) [Epoch 6/10] step 1700/13933 (6.4 it/s) [Epoch 6/10] step 1750/13933 (6.4 it/s) [Epoch 6/10] step 1800/13933 (6.4 it/s) [Epoch 6/10] step 1850/13933 (6.4 it/s) [Epoch 6/10] step 1900/13933 (6.4 it/s) [Epoch 6/10] step 1950/13933 (6.4 it/s) [Epoch 6/10] step 2000/13933 (6.4 it/s) [Epoch 6/10] step 2050/13933 (6.4 it/s) [Epoch 6/10] step 2100/13933 (6.4 it/s) [Epoch 6/10] step 2150/13933 (6.4 it/s) [Epoch 6/10] step 2200/13933 (6.4 it/s) [Epoch 6/10] step 2250/13933 (6.4 it/s) [Epoch 6/10] step 2300/13933 (6.4 it/s) [Epoch 6/10] step 2350/13933 (6.4 it/s) [Epoch 6/10] step 2400/13933 (6.4 it/s) [Epoch 6/10] step 2450/13933 (6.4 it/s) [Epoch 6/10] step 2500/13933 (6.4 it/s) [Epoch 6/10] step 2550/13933 (6.4 it/s) [Epoch 6/10] step 2600/13933 (6.4 it/s) [Epoch 6/10] step 2650/13933 (6.4 it/s) [Epoch 6/10] step 2700/13933 (6.4 it/s) [Epoch 6/10] step 2750/13933 (6.4 it/s) [Epoch 6/10] step 2800/13933 (6.4 it/s) [Epoch 6/10] step 2850/13933 (6.4 it/s) [Epoch 6/10] step 2900/13933 (6.4 it/s) [Epoch 6/10] step 2950/13933 (6.4 it/s) [Epoch 6/10] step 3000/13933 (6.4 it/s) [Epoch 6/10] step 3050/13933 (6.4 it/s) [Epoch 6/10] step 3100/13933 (6.4 it/s) [Epoch 6/10] step 3150/13933 (6.4 it/s) [Epoch 6/10] step 3200/13933 (6.4 it/s) [Epoch 6/10] step 3250/13933 (6.4 it/s) [Epoch 6/10] step 3300/13933 (6.4 it/s) [Epoch 6/10] step 3350/13933 (6.4 it/s) [Epoch 6/10] step 3400/13933 (6.4 it/s) [Epoch 6/10] step 3450/13933 (6.4 it/s) [Epoch 6/10] step 3500/13933 (6.4 it/s) [Epoch 6/10] step 3550/13933 (6.4 it/s) [Epoch 6/10] step 3600/13933 (6.4 it/s) [Epoch 6/10] step 3650/13933 (6.4 it/s) [Epoch 6/10] step 3700/13933 (6.4 it/s) [Epoch 6/10] step 3750/13933 (6.4 it/s) [Epoch 6/10] step 3800/13933 (6.4 it/s) [Epoch 6/10] step 3850/13933 (6.4 it/s) [Epoch 6/10] step 3900/13933 (6.4 it/s) [Epoch 6/10] step 3950/13933 (6.4 it/s) [Epoch 6/10] step 4000/13933 (6.4 it/s) [Epoch 6/10] step 4050/13933 (6.4 it/s) [Epoch 6/10] step 4100/13933 (6.5 it/s) [Epoch 6/10] step 4150/13933 (6.5 it/s) [Epoch 6/10] step 4200/13933 (6.5 it/s) [Epoch 6/10] step 4250/13933 (6.5 it/s) [Epoch 6/10] step 4300/13933 (6.5 it/s) [Epoch 6/10] step 4350/13933 (6.5 it/s) [Epoch 6/10] step 4400/13933 (6.5 it/s) [Epoch 6/10] step 4450/13933 (6.5 it/s) [Epoch 6/10] step 4500/13933 (6.5 it/s) [Epoch 6/10] step 4550/13933 (6.5 it/s) [Epoch 6/10] step 4600/13933 (6.5 it/s) [Epoch 6/10] step 4650/13933 (6.5 it/s) [Epoch 6/10] step 4700/13933 (6.5 it/s) [Epoch 6/10] step 4750/13933 (6.5 it/s) [Epoch 6/10] step 4800/13933 (6.5 it/s) [Epoch 6/10] step 4850/13933 (6.5 it/s) [Epoch 6/10] step 4900/13933 (6.5 it/s) [Epoch 6/10] step 4950/13933 (6.5 it/s) [Epoch 6/10] step 5000/13933 (6.5 it/s) [Epoch 6/10] step 5050/13933 (6.5 it/s) [Epoch 6/10] step 5100/13933 (6.5 it/s) [Epoch 6/10] step 5150/13933 (6.4 it/s) [Epoch 6/10] step 5200/13933 (6.4 it/s) [Epoch 6/10] step 5250/13933 (6.4 it/s) [Epoch 6/10] step 5300/13933 (6.3 it/s) [Epoch 6/10] step 5350/13933 (6.3 it/s) [Epoch 6/10] step 5400/13933 (6.3 it/s) [Epoch 6/10] step 5450/13933 (6.3 it/s) [Epoch 6/10] step 5500/13933 (6.3 it/s) [Epoch 6/10] step 5550/13933 (6.3 it/s) [Epoch 6/10] step 5600/13933 (6.2 it/s) [Epoch 6/10] step 5650/13933 (6.2 it/s) [Epoch 6/10] step 5700/13933 (6.2 it/s) [Epoch 6/10] step 5750/13933 (6.2 it/s) [Epoch 6/10] step 5800/13933 (6.2 it/s) [Epoch 6/10] step 5850/13933 (6.2 it/s) [Epoch 6/10] step 5900/13933 (6.2 it/s) [Epoch 6/10] step 5950/13933 (6.2 it/s) [Epoch 6/10] step 6000/13933 (6.2 it/s) [Epoch 6/10] step 6050/13933 (6.1 it/s) [Epoch 6/10] step 6100/13933 (6.1 it/s) [Epoch 6/10] step 6150/13933 (6.1 it/s) [Epoch 6/10] step 6200/13933 (6.1 it/s) [Epoch 6/10] step 6250/13933 (6.1 it/s) [Epoch 6/10] step 6300/13933 (6.1 it/s) [Epoch 6/10] step 6350/13933 (6.1 it/s) [Epoch 6/10] step 6400/13933 (6.1 it/s) [Epoch 6/10] step 6450/13933 (6.1 it/s) [Epoch 6/10] step 6500/13933 (6.1 it/s) [Epoch 6/10] step 6550/13933 (6.0 it/s) [Epoch 6/10] step 6600/13933 (6.0 it/s) [Epoch 6/10] step 6650/13933 (6.0 it/s) [Epoch 6/10] step 6700/13933 (6.0 it/s) [Epoch 6/10] step 6750/13933 (6.0 it/s) [Epoch 6/10] step 6800/13933 (6.1 it/s) [Epoch 6/10] step 6850/13933 (6.0 it/s) [Epoch 6/10] step 6900/13933 (6.0 it/s) [Epoch 6/10] step 6950/13933 (6.0 it/s) [Epoch 6/10] step 7000/13933 (6.0 it/s) [Epoch 6/10] step 7050/13933 (6.0 it/s) [Epoch 6/10] step 7100/13933 (6.0 it/s) [Epoch 6/10] step 7150/13933 (6.0 it/s) [Epoch 6/10] step 7200/13933 (6.0 it/s) [Epoch 6/10] step 7250/13933 (6.0 it/s) [Epoch 6/10] step 7300/13933 (6.0 it/s) [Epoch 6/10] step 7350/13933 (6.0 it/s) [Epoch 6/10] step 7400/13933 (5.9 it/s) [Epoch 6/10] step 7450/13933 (5.9 it/s) [Epoch 6/10] step 7500/13933 (5.9 it/s) [Epoch 6/10] step 7550/13933 (5.9 it/s) [Epoch 6/10] step 7600/13933 (5.9 it/s) [Epoch 6/10] step 7650/13933 (5.8 it/s) [Epoch 6/10] step 7700/13933 (5.8 it/s) [Epoch 6/10] step 7750/13933 (5.8 it/s) [Epoch 6/10] step 7800/13933 (5.8 it/s) [Epoch 6/10] step 7850/13933 (5.7 it/s) [Epoch 6/10] step 7900/13933 (5.7 it/s) [Epoch 6/10] step 7950/13933 (5.7 it/s) [Epoch 6/10] step 8000/13933 (5.6 it/s) [Epoch 6/10] step 8050/13933 (5.6 it/s) [Epoch 6/10] step 8100/13933 (5.6 it/s) [Epoch 6/10] step 8150/13933 (5.6 it/s) [Epoch 6/10] step 8200/13933 (5.6 it/s) [Epoch 6/10] step 8250/13933 (5.6 it/s) [Epoch 6/10] step 8300/13933 (5.6 it/s) [Epoch 6/10] step 8350/13933 (5.5 it/s) [Epoch 6/10] step 8400/13933 (5.5 it/s) [Epoch 6/10] step 8450/13933 (5.5 it/s) [Epoch 6/10] step 8500/13933 (5.5 it/s) [Epoch 6/10] step 8550/13933 (5.5 it/s) [Epoch 6/10] step 8600/13933 (5.5 it/s) [Epoch 6/10] step 8650/13933 (5.5 it/s) [Epoch 6/10] step 8700/13933 (5.5 it/s) [Epoch 6/10] step 8750/13933 (5.5 it/s) [Epoch 6/10] step 8800/13933 (5.5 it/s) [Epoch 6/10] step 8850/13933 (5.4 it/s) [Epoch 6/10] step 8900/13933 (5.4 it/s) [Epoch 6/10] step 8950/13933 (5.4 it/s) [Epoch 6/10] step 9000/13933 (5.4 it/s) [Epoch 6/10] step 9050/13933 (5.4 it/s) [Epoch 6/10] step 9100/13933 (5.4 it/s) [Epoch 6/10] step 9150/13933 (5.4 it/s) [Epoch 6/10] step 9200/13933 (5.4 it/s) [Epoch 6/10] step 9250/13933 (5.4 it/s) [Epoch 6/10] step 9300/13933 (5.4 it/s) [Epoch 6/10] step 9350/13933 (5.4 it/s) [Epoch 6/10] step 9400/13933 (5.4 it/s) [Epoch 6/10] step 9450/13933 (5.4 it/s) [Epoch 6/10] step 9500/13933 (5.4 it/s) [Epoch 6/10] step 9550/13933 (5.3 it/s) [Epoch 6/10] step 9600/13933 (5.3 it/s) [Epoch 6/10] step 9650/13933 (5.3 it/s) [Epoch 6/10] step 9700/13933 (5.3 it/s) [Epoch 6/10] step 9750/13933 (5.3 it/s) [Epoch 6/10] step 9800/13933 (5.3 it/s) [Epoch 6/10] step 9850/13933 (5.3 it/s) [Epoch 6/10] step 9900/13933 (5.3 it/s) [Epoch 6/10] step 9950/13933 (5.3 it/s) [Epoch 6/10] step 10000/13933 (5.3 it/s) [Epoch 6/10] step 10050/13933 (5.3 it/s) [Epoch 6/10] step 10100/13933 (5.4 it/s) [Epoch 6/10] step 10150/13933 (5.4 it/s) [Epoch 6/10] step 10200/13933 (5.4 it/s) [Epoch 6/10] step 10250/13933 (5.4 it/s) [Epoch 6/10] step 10300/13933 (5.4 it/s) [Epoch 6/10] step 10350/13933 (5.4 it/s) [Epoch 6/10] step 10400/13933 (5.4 it/s) [Epoch 6/10] step 10450/13933 (5.4 it/s) [Epoch 6/10] step 10500/13933 (5.4 it/s) [Epoch 6/10] step 10550/13933 (5.4 it/s) [Epoch 6/10] step 10600/13933 (5.4 it/s) [Epoch 6/10] step 10650/13933 (5.4 it/s) [Epoch 6/10] step 10700/13933 (5.4 it/s) [Epoch 6/10] step 10750/13933 (5.4 it/s) [Epoch 6/10] step 10800/13933 (5.4 it/s) [Epoch 6/10] step 10850/13933 (5.4 it/s) [Epoch 6/10] step 10900/13933 (5.4 it/s) [Epoch 6/10] step 10950/13933 (5.5 it/s) [Epoch 6/10] step 11000/13933 (5.5 it/s) [Epoch 6/10] step 11050/13933 (5.5 it/s) [Epoch 6/10] step 11100/13933 (5.5 it/s) [Epoch 6/10] step 11150/13933 (5.5 it/s) [Epoch 6/10] step 11200/13933 (5.5 it/s) [Epoch 6/10] step 11250/13933 (5.5 it/s) [Epoch 6/10] step 11300/13933 (5.5 it/s) [Epoch 6/10] step 11350/13933 (5.5 it/s) [Epoch 6/10] step 11400/13933 (5.5 it/s) [Epoch 6/10] step 11450/13933 (5.5 it/s) [Epoch 6/10] step 11500/13933 (5.5 it/s) [Epoch 6/10] step 11550/13933 (5.5 it/s) [Epoch 6/10] step 11600/13933 (5.5 it/s) [Epoch 6/10] step 11650/13933 (5.5 it/s) [Epoch 6/10] step 11700/13933 (5.5 it/s) [Epoch 6/10] step 11750/13933 (5.5 it/s) [Epoch 6/10] step 11800/13933 (5.5 it/s) [Epoch 6/10] step 11850/13933 (5.6 it/s) [Epoch 6/10] step 11900/13933 (5.6 it/s) [Epoch 6/10] step 11950/13933 (5.6 it/s) [Epoch 6/10] step 12000/13933 (5.6 it/s) [Epoch 6/10] step 12050/13933 (5.6 it/s) [Epoch 6/10] step 12100/13933 (5.6 it/s) [Epoch 6/10] step 12150/13933 (5.6 it/s) [Epoch 6/10] step 12200/13933 (5.6 it/s) [Epoch 6/10] step 12250/13933 (5.6 it/s) [Epoch 6/10] step 12300/13933 (5.6 it/s) [Epoch 6/10] step 12350/13933 (5.6 it/s) [Epoch 6/10] step 12400/13933 (5.6 it/s) [Epoch 6/10] step 12450/13933 (5.6 it/s) [Epoch 6/10] step 12500/13933 (5.6 it/s) [Epoch 6/10] step 12550/13933 (5.6 it/s) [Epoch 6/10] step 12600/13933 (5.6 it/s) [Epoch 6/10] step 12650/13933 (5.6 it/s) [Epoch 6/10] step 12700/13933 (5.6 it/s) [Epoch 6/10] step 12750/13933 (5.6 it/s) [Epoch 6/10] step 12800/13933 (5.6 it/s) [Epoch 6/10] step 12850/13933 (5.6 it/s) [Epoch 6/10] step 12900/13933 (5.6 it/s) [Epoch 6/10] step 12950/13933 (5.7 it/s) [Epoch 6/10] step 13000/13933 (5.7 it/s) [Epoch 6/10] step 13050/13933 (5.7 it/s) [Epoch 6/10] step 13100/13933 (5.7 it/s) [Epoch 6/10] step 13150/13933 (5.7 it/s) [Epoch 6/10] step 13200/13933 (5.7 it/s) [Epoch 6/10] step 13250/13933 (5.7 it/s) [Epoch 6/10] step 13300/13933 (5.7 it/s) [Epoch 6/10] step 13350/13933 (5.7 it/s) [Epoch 6/10] step 13400/13933 (5.7 it/s) [Epoch 6/10] step 13450/13933 (5.7 it/s) [Epoch 6/10] step 13500/13933 (5.7 it/s) [Epoch 6/10] step 13550/13933 (5.7 it/s) [Epoch 6/10] step 13600/13933 (5.7 it/s) [Epoch 6/10] step 13650/13933 (5.7 it/s) [Epoch 6/10] step 13700/13933 (5.7 it/s) [Epoch 6/10] step 13750/13933 (5.7 it/s) [Epoch 6/10] step 13800/13933 (5.7 it/s) [Epoch 6/10] step 13850/13933 (5.7 it/s) [Epoch 6/10] step 13900/13933 (5.7 it/s) 16:47:25 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.08552702516317368 | | fit/pred_loss | 0.008378587663173676 | | fit/sigreg_loss | 0.859375 | | hardware/cpu_percent | 25.0 | | hardware/disk_read_mb_s | 1906.3260498046875 | | hardware/disk_write_mb_s | 0.6968109607696533 | | hardware/net_recv_mb_s | 0.0002642786712385714 | | hardware/net_sent_mb_s | 0.0003889815998263657 | | hardware/ram_used_gb | 200.0858154296875 | | hardware/ram_used_pct | 39.70000076293945 | | hparams/lr_model_opt | 1.0498534720682073e-05 | | validate/loss | 0.1268523633480072 | | validate/loss_epoch | 0.1268523633480072 | | validate/pred_loss | 0.0041648512706160545 | | validate/pred_loss_epoch | 0.0041648512706160545 | | validate/sigreg_loss | 1.3631764650344849 | | validate/sigreg_loss_epoch | 1.3631764650344849 | +----------------------------+------------------------+ 16:47:25 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_007.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 16:47:25 | INFO | checkpoint_~| Checking for non PyTorch modules to save 16:47:25 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 16:47:26 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.33s write=0.14s rename=0.2ms) [Epoch 6/10] done in 2654.0s 16:47:26 | INFO | checkpoint_~| Checking for non PyTorch modules to save 16:47:26 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=6-step=97531.ckpt 16:47:26 | INFO | atomic_chec~| [atomic_save] ✓ epoch=6-step=97531.ckpt saved (206.8 MiB, serialise=0.30s write=0.13s rename=0.1ms) 16:47:26 | INFO | checkpoint_~| Checking for non PyTorch modules to save 16:47:26 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 16:47:26 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.26s write=0.11s rename=1.3ms) [Epoch 7/10] step 50/13933 (5.8 it/s) [Epoch 7/10] step 100/13933 (6.3 it/s) [Epoch 7/10] step 150/13933 (6.4 it/s) [Epoch 7/10] step 200/13933 (6.6 it/s) [Epoch 7/10] step 250/13933 (6.7 it/s) [Epoch 7/10] step 300/13933 (6.8 it/s) [Epoch 7/10] step 350/13933 (6.8 it/s) [Epoch 7/10] step 400/13933 (6.8 it/s) [Epoch 7/10] step 450/13933 (6.8 it/s) [Epoch 7/10] step 500/13933 (6.9 it/s) [Epoch 7/10] step 550/13933 (6.9 it/s) [Epoch 7/10] step 600/13933 (6.9 it/s) [Epoch 7/10] step 650/13933 (6.9 it/s) [Epoch 7/10] step 700/13933 (6.9 it/s) [Epoch 7/10] step 750/13933 (7.0 it/s) [Epoch 7/10] step 800/13933 (7.0 it/s) [Epoch 7/10] step 850/13933 (7.0 it/s) [Epoch 7/10] step 900/13933 (7.0 it/s) [Epoch 7/10] step 950/13933 (7.0 it/s) [Epoch 7/10] step 1000/13933 (7.0 it/s) [Epoch 7/10] step 1050/13933 (7.0 it/s) [Epoch 7/10] step 1100/13933 (7.0 it/s) [Epoch 7/10] step 1150/13933 (7.0 it/s) [Epoch 7/10] step 1200/13933 (7.0 it/s) [Epoch 7/10] step 1250/13933 (7.0 it/s) [Epoch 7/10] step 1300/13933 (7.0 it/s) [Epoch 7/10] step 1350/13933 (7.0 it/s) [Epoch 7/10] step 1400/13933 (7.0 it/s) [Epoch 7/10] step 1450/13933 (7.0 it/s) [Epoch 7/10] step 1500/13933 (7.0 it/s) [Epoch 7/10] step 1550/13933 (7.0 it/s) [Epoch 7/10] step 1600/13933 (7.0 it/s) [Epoch 7/10] step 1650/13933 (7.0 it/s) [Epoch 7/10] step 1700/13933 (7.0 it/s) [Epoch 7/10] step 1750/13933 (7.0 it/s) [Epoch 7/10] step 1800/13933 (7.0 it/s) [Epoch 7/10] step 1850/13933 (7.1 it/s) [Epoch 7/10] step 1900/13933 (7.1 it/s) [Epoch 7/10] step 1950/13933 (7.1 it/s) [Epoch 7/10] step 2000/13933 (7.0 it/s) [Epoch 7/10] step 2050/13933 (7.1 it/s) [Epoch 7/10] step 2100/13933 (7.1 it/s) [Epoch 7/10] step 2150/13933 (7.1 it/s) [Epoch 7/10] step 2200/13933 (7.1 it/s) [Epoch 7/10] step 2250/13933 (7.1 it/s) [Epoch 7/10] step 2300/13933 (7.1 it/s) [Epoch 7/10] step 2350/13933 (7.1 it/s) [Epoch 7/10] step 2400/13933 (7.1 it/s) [Epoch 7/10] step 2450/13933 (7.1 it/s) [Epoch 7/10] step 2500/13933 (7.1 it/s) [Epoch 7/10] step 2550/13933 (7.1 it/s) [Epoch 7/10] step 2600/13933 (7.1 it/s) [Epoch 7/10] step 2650/13933 (7.1 it/s) [Epoch 7/10] step 2700/13933 (7.1 it/s) [Epoch 7/10] step 2750/13933 (7.1 it/s) [Epoch 7/10] step 2800/13933 (7.1 it/s) [Epoch 7/10] step 2850/13933 (7.1 it/s) [Epoch 7/10] step 2900/13933 (7.1 it/s) [Epoch 7/10] step 2950/13933 (7.1 it/s) [Epoch 7/10] step 3000/13933 (7.1 it/s) [Epoch 7/10] step 3050/13933 (7.1 it/s) [Epoch 7/10] step 3100/13933 (7.1 it/s) [Epoch 7/10] step 3150/13933 (7.1 it/s) [Epoch 7/10] step 3200/13933 (7.1 it/s) [Epoch 7/10] step 3250/13933 (7.1 it/s) [Epoch 7/10] step 3300/13933 (7.1 it/s) [Epoch 7/10] step 3350/13933 (7.1 it/s) [Epoch 7/10] step 3400/13933 (7.1 it/s) [Epoch 7/10] step 3450/13933 (7.1 it/s) [Epoch 7/10] step 3500/13933 (7.1 it/s) [Epoch 7/10] step 3550/13933 (7.1 it/s) [Epoch 7/10] step 3600/13933 (7.1 it/s) [Epoch 7/10] step 3650/13933 (7.1 it/s) [Epoch 7/10] step 3700/13933 (7.1 it/s) [Epoch 7/10] step 3750/13933 (7.1 it/s) [Epoch 7/10] step 3800/13933 (7.1 it/s) [Epoch 7/10] step 3850/13933 (7.1 it/s) [Epoch 7/10] step 3900/13933 (7.1 it/s) [Epoch 7/10] step 3950/13933 (7.1 it/s) [Epoch 7/10] step 4000/13933 (7.1 it/s) [Epoch 7/10] step 4050/13933 (7.1 it/s) [Epoch 7/10] step 4100/13933 (7.1 it/s) [Epoch 7/10] step 4150/13933 (7.1 it/s) [Epoch 7/10] step 4200/13933 (7.1 it/s) [Epoch 7/10] step 4250/13933 (7.1 it/s) [Epoch 7/10] step 4300/13933 (7.1 it/s) [Epoch 7/10] step 4350/13933 (7.1 it/s) [Epoch 7/10] step 4400/13933 (7.1 it/s) [Epoch 7/10] step 4450/13933 (7.1 it/s) [Epoch 7/10] step 4500/13933 (7.1 it/s) [Epoch 7/10] step 4550/13933 (7.1 it/s) [Epoch 7/10] step 4600/13933 (7.1 it/s) [Epoch 7/10] step 4650/13933 (7.1 it/s) [Epoch 7/10] step 4700/13933 (7.1 it/s) [Epoch 7/10] step 4750/13933 (7.1 it/s) [Epoch 7/10] step 4800/13933 (7.1 it/s) [Epoch 7/10] step 4850/13933 (7.1 it/s) [Epoch 7/10] step 4900/13933 (7.1 it/s) [Epoch 7/10] step 4950/13933 (7.1 it/s) [Epoch 7/10] step 5000/13933 (7.1 it/s) [Epoch 7/10] step 5050/13933 (7.1 it/s) [Epoch 7/10] step 5100/13933 (7.1 it/s) [Epoch 7/10] step 5150/13933 (7.1 it/s) [Epoch 7/10] step 5200/13933 (7.1 it/s) [Epoch 7/10] step 5250/13933 (7.1 it/s) [Epoch 7/10] step 5300/13933 (7.1 it/s) [Epoch 7/10] step 5350/13933 (7.1 it/s) [Epoch 7/10] step 5400/13933 (7.1 it/s) [Epoch 7/10] step 5450/13933 (7.1 it/s) [Epoch 7/10] step 5500/13933 (7.1 it/s) [Epoch 7/10] step 5550/13933 (7.1 it/s) [Epoch 7/10] step 5600/13933 (7.1 it/s) [Epoch 7/10] step 5650/13933 (7.1 it/s) [Epoch 7/10] step 5700/13933 (7.1 it/s) [Epoch 7/10] step 5750/13933 (7.1 it/s) [Epoch 7/10] step 5800/13933 (7.1 it/s) [Epoch 7/10] step 5850/13933 (7.1 it/s) [Epoch 7/10] step 5900/13933 (7.1 it/s) [Epoch 7/10] step 5950/13933 (7.1 it/s) [Epoch 7/10] step 6000/13933 (7.1 it/s) [Epoch 7/10] step 6050/13933 (7.1 it/s) [Epoch 7/10] step 6100/13933 (7.1 it/s) [Epoch 7/10] step 6150/13933 (7.1 it/s) [Epoch 7/10] step 6200/13933 (7.1 it/s) [Epoch 7/10] step 6250/13933 (7.1 it/s) [Epoch 7/10] step 6300/13933 (7.1 it/s) [Epoch 7/10] step 6350/13933 (7.1 it/s) [Epoch 7/10] step 6400/13933 (7.1 it/s) [Epoch 7/10] step 6450/13933 (7.1 it/s) [Epoch 7/10] step 6500/13933 (7.1 it/s) [Epoch 7/10] step 6550/13933 (7.1 it/s) [Epoch 7/10] step 6600/13933 (7.1 it/s) [Epoch 7/10] step 6650/13933 (7.1 it/s) [Epoch 7/10] step 6700/13933 (7.1 it/s) [Epoch 7/10] step 6750/13933 (7.1 it/s) [Epoch 7/10] step 6800/13933 (7.1 it/s) [Epoch 7/10] step 6850/13933 (7.1 it/s) [Epoch 7/10] step 6900/13933 (7.1 it/s) [Epoch 7/10] step 6950/13933 (7.1 it/s) [Epoch 7/10] step 7000/13933 (7.1 it/s) [Epoch 7/10] step 7050/13933 (7.1 it/s) [Epoch 7/10] step 7100/13933 (7.1 it/s) [Epoch 7/10] step 7150/13933 (7.1 it/s) [Epoch 7/10] step 7200/13933 (7.1 it/s) [Epoch 7/10] step 7250/13933 (7.1 it/s) [Epoch 7/10] step 7300/13933 (7.0 it/s) [Epoch 7/10] step 7350/13933 (7.0 it/s) [Epoch 7/10] step 7400/13933 (7.0 it/s) [Epoch 7/10] step 7450/13933 (7.0 it/s) [Epoch 7/10] step 7500/13933 (7.0 it/s) [Epoch 7/10] step 7550/13933 (7.0 it/s) [Epoch 7/10] step 7600/13933 (7.0 it/s) [Epoch 7/10] step 7650/13933 (7.0 it/s) [Epoch 7/10] step 7700/13933 (7.0 it/s) [Epoch 7/10] step 7750/13933 (7.0 it/s) [Epoch 7/10] step 7800/13933 (7.0 it/s) [Epoch 7/10] step 7850/13933 (7.0 it/s) [Epoch 7/10] step 7900/13933 (7.0 it/s) [Epoch 7/10] step 7950/13933 (7.0 it/s) [Epoch 7/10] step 8000/13933 (7.0 it/s) [Epoch 7/10] step 8050/13933 (7.0 it/s) [Epoch 7/10] step 8100/13933 (7.0 it/s) [Epoch 7/10] step 8150/13933 (7.0 it/s) [Epoch 7/10] step 8200/13933 (7.0 it/s) [Epoch 7/10] step 8250/13933 (7.0 it/s) [Epoch 7/10] step 8300/13933 (7.0 it/s) [Epoch 7/10] step 8350/13933 (7.0 it/s) [Epoch 7/10] step 8400/13933 (7.0 it/s) [Epoch 7/10] step 8450/13933 (7.0 it/s) [Epoch 7/10] step 8500/13933 (7.0 it/s) [Epoch 7/10] step 8550/13933 (7.0 it/s) [Epoch 7/10] step 8600/13933 (7.0 it/s) [Epoch 7/10] step 8650/13933 (7.0 it/s) [Epoch 7/10] step 8700/13933 (7.0 it/s) [Epoch 7/10] step 8750/13933 (7.0 it/s) [Epoch 7/10] step 8800/13933 (7.0 it/s) [Epoch 7/10] step 8850/13933 (7.0 it/s) [Epoch 7/10] step 8900/13933 (7.0 it/s) [Epoch 7/10] step 8950/13933 (7.0 it/s) [Epoch 7/10] step 9000/13933 (7.0 it/s) [Epoch 7/10] step 9050/13933 (7.0 it/s) [Epoch 7/10] step 9100/13933 (7.0 it/s) [Epoch 7/10] step 9150/13933 (7.0 it/s) [Epoch 7/10] step 9200/13933 (7.0 it/s) [Epoch 7/10] step 9250/13933 (7.0 it/s) [Epoch 7/10] step 9300/13933 (7.0 it/s) [Epoch 7/10] step 9350/13933 (7.0 it/s) [Epoch 7/10] step 9400/13933 (7.0 it/s) [Epoch 7/10] step 9450/13933 (7.0 it/s) [Epoch 7/10] step 9500/13933 (7.0 it/s) [Epoch 7/10] step 9550/13933 (7.0 it/s) [Epoch 7/10] step 9600/13933 (7.0 it/s) [Epoch 7/10] step 9650/13933 (7.0 it/s) [Epoch 7/10] step 9700/13933 (7.0 it/s) [Epoch 7/10] step 9750/13933 (7.0 it/s) [Epoch 7/10] step 9800/13933 (7.0 it/s) [Epoch 7/10] step 9850/13933 (6.9 it/s) [Epoch 7/10] step 9900/13933 (6.9 it/s) [Epoch 7/10] step 9950/13933 (6.9 it/s) [Epoch 7/10] step 10000/13933 (6.9 it/s) [Epoch 7/10] step 10050/13933 (6.9 it/s) [Epoch 7/10] step 10100/13933 (6.9 it/s) [Epoch 7/10] step 10150/13933 (6.9 it/s) [Epoch 7/10] step 10200/13933 (6.9 it/s) [Epoch 7/10] step 10250/13933 (6.9 it/s) [Epoch 7/10] step 10300/13933 (6.9 it/s) [Epoch 7/10] step 10350/13933 (6.9 it/s) [Epoch 7/10] step 10400/13933 (6.9 it/s) [Epoch 7/10] step 10450/13933 (6.9 it/s) [Epoch 7/10] step 10500/13933 (6.9 it/s) [Epoch 7/10] step 10550/13933 (6.9 it/s) [Epoch 7/10] step 10600/13933 (6.9 it/s) [Epoch 7/10] step 10650/13933 (6.9 it/s) [Epoch 7/10] step 10700/13933 (6.9 it/s) [Epoch 7/10] step 10750/13933 (6.9 it/s) [Epoch 7/10] step 10800/13933 (6.9 it/s) [Epoch 7/10] step 10850/13933 (6.9 it/s) [Epoch 7/10] step 10900/13933 (6.9 it/s) [Epoch 7/10] step 10950/13933 (6.9 it/s) [Epoch 7/10] step 11000/13933 (6.9 it/s) [Epoch 7/10] step 11050/13933 (6.9 it/s) [Epoch 7/10] step 11100/13933 (6.9 it/s) [Epoch 7/10] step 11150/13933 (6.9 it/s) [Epoch 7/10] step 11200/13933 (7.0 it/s) [Epoch 7/10] step 11250/13933 (7.0 it/s) [Epoch 7/10] step 11300/13933 (7.0 it/s) [Epoch 7/10] step 11350/13933 (7.0 it/s) [Epoch 7/10] step 11400/13933 (7.0 it/s) [Epoch 7/10] step 11450/13933 (7.0 it/s) [Epoch 7/10] step 11500/13933 (7.0 it/s) [Epoch 7/10] step 11550/13933 (7.0 it/s) [Epoch 7/10] step 11600/13933 (7.0 it/s) [Epoch 7/10] step 11650/13933 (7.0 it/s) [Epoch 7/10] step 11700/13933 (7.0 it/s) [Epoch 7/10] step 11750/13933 (7.0 it/s) [Epoch 7/10] step 11800/13933 (7.0 it/s) [Epoch 7/10] step 11850/13933 (7.0 it/s) [Epoch 7/10] step 11900/13933 (7.0 it/s) [Epoch 7/10] step 11950/13933 (7.0 it/s) [Epoch 7/10] step 12000/13933 (7.0 it/s) [Epoch 7/10] step 12050/13933 (7.0 it/s) [Epoch 7/10] step 12100/13933 (7.0 it/s) [Epoch 7/10] step 12150/13933 (7.0 it/s) [Epoch 7/10] step 12200/13933 (7.0 it/s) [Epoch 7/10] step 12250/13933 (7.0 it/s) [Epoch 7/10] step 12300/13933 (7.0 it/s) [Epoch 7/10] step 12350/13933 (7.0 it/s) [Epoch 7/10] step 12400/13933 (7.0 it/s) [Epoch 7/10] step 12450/13933 (7.0 it/s) [Epoch 7/10] step 12500/13933 (7.0 it/s) [Epoch 7/10] step 12550/13933 (7.0 it/s) [Epoch 7/10] step 12600/13933 (7.0 it/s) [Epoch 7/10] step 12650/13933 (7.0 it/s) [Epoch 7/10] step 12700/13933 (7.0 it/s) [Epoch 7/10] step 12750/13933 (7.0 it/s) [Epoch 7/10] step 12800/13933 (7.0 it/s) [Epoch 7/10] step 12850/13933 (7.0 it/s) [Epoch 7/10] step 12900/13933 (7.0 it/s) [Epoch 7/10] step 12950/13933 (7.0 it/s) [Epoch 7/10] step 13000/13933 (7.0 it/s) [Epoch 7/10] step 13050/13933 (7.0 it/s) [Epoch 7/10] step 13100/13933 (7.0 it/s) [Epoch 7/10] step 13150/13933 (7.0 it/s) [Epoch 7/10] step 13200/13933 (7.0 it/s) [Epoch 7/10] step 13250/13933 (7.0 it/s) [Epoch 7/10] step 13300/13933 (7.0 it/s) [Epoch 7/10] step 13350/13933 (7.0 it/s) [Epoch 7/10] step 13400/13933 (7.0 it/s) [Epoch 7/10] step 13450/13933 (7.0 it/s) [Epoch 7/10] step 13500/13933 (7.0 it/s) [Epoch 7/10] step 13550/13933 (7.0 it/s) [Epoch 7/10] step 13600/13933 (7.0 it/s) [Epoch 7/10] step 13650/13933 (7.0 it/s) [Epoch 7/10] step 13700/13933 (7.0 it/s) [Epoch 7/10] step 13750/13933 (7.0 it/s) [Epoch 7/10] step 13800/13933 (6.9 it/s) [Epoch 7/10] step 13850/13933 (6.9 it/s) [Epoch 7/10] step 13900/13933 (6.9 it/s) 17:24:55 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.08774025738239288 | | fit/pred_loss | 0.008150417357683182 | | fit/sigreg_loss | 0.8828125 | | hardware/cpu_percent | 29.100000381469727 | | hardware/disk_read_mb_s | 1755.2518310546875 | | hardware/disk_write_mb_s | 0.6809386014938354 | | hardware/net_recv_mb_s | 0.0001836463634390384 | | hardware/net_sent_mb_s | 0.00027136944117955863 | | hardware/ram_used_gb | 191.90809631347656 | | hardware/ram_used_pct | 38.099998474121094 | | hparams/lr_model_opt | 4.868223186349496e-06 | | validate/loss | 0.12397567927837372 | | validate/loss_epoch | 0.12397567927837372 | | validate/pred_loss | 0.0034982794895768166 | | validate/pred_loss_epoch | 0.0034982794895768166 | | validate/sigreg_loss | 1.3387010097503662 | | validate/sigreg_loss_epoch | 1.3387010097503662 | +----------------------------+------------------------+ 17:24:55 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_008.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 17:24:55 | INFO | checkpoint_~| Checking for non PyTorch modules to save 17:24:55 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 17:24:56 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.29s write=0.16s rename=0.1ms) [Epoch 7/10] done in 2249.6s 17:24:56 | INFO | checkpoint_~| Checking for non PyTorch modules to save 17:24:56 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=7-step=111464.ckpt 17:24:56 | INFO | atomic_chec~| [atomic_save] ✓ epoch=7-step=111464.ckpt saved (206.8 MiB, serialise=0.27s write=0.18s rename=0.1ms) 17:24:56 | INFO | checkpoint_~| Checking for non PyTorch modules to save 17:24:56 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 17:24:57 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.26s write=0.15s rename=0.1ms) [Epoch 8/10] step 50/13933 (5.7 it/s) [Epoch 8/10] step 100/13933 (6.3 it/s) [Epoch 8/10] step 150/13933 (6.6 it/s) [Epoch 8/10] step 200/13933 (6.7 it/s) [Epoch 8/10] step 250/13933 (6.8 it/s) [Epoch 8/10] step 300/13933 (6.9 it/s) [Epoch 8/10] step 350/13933 (6.9 it/s) [Epoch 8/10] step 400/13933 (7.0 it/s) [Epoch 8/10] step 450/13933 (7.0 it/s) [Epoch 8/10] step 500/13933 (7.0 it/s) [Epoch 8/10] step 550/13933 (7.0 it/s) [Epoch 8/10] step 600/13933 (7.0 it/s) [Epoch 8/10] step 650/13933 (7.0 it/s) [Epoch 8/10] step 700/13933 (7.0 it/s) [Epoch 8/10] step 750/13933 (7.0 it/s) [Epoch 8/10] step 800/13933 (7.0 it/s) [Epoch 8/10] step 850/13933 (7.0 it/s) [Epoch 8/10] step 900/13933 (7.0 it/s) [Epoch 8/10] step 950/13933 (7.0 it/s) [Epoch 8/10] step 1000/13933 (7.0 it/s) [Epoch 8/10] step 1050/13933 (7.0 it/s) [Epoch 8/10] step 1100/13933 (7.0 it/s) [Epoch 8/10] step 1150/13933 (7.0 it/s) [Epoch 8/10] step 1200/13933 (7.0 it/s) [Epoch 8/10] step 1250/13933 (7.0 it/s) [Epoch 8/10] step 1300/13933 (7.0 it/s) [Epoch 8/10] step 1350/13933 (7.0 it/s) [Epoch 8/10] step 1400/13933 (7.0 it/s) [Epoch 8/10] step 1450/13933 (7.0 it/s) [Epoch 8/10] step 1500/13933 (7.0 it/s) [Epoch 8/10] step 1550/13933 (7.0 it/s) [Epoch 8/10] step 1600/13933 (7.0 it/s) [Epoch 8/10] step 1650/13933 (7.0 it/s) [Epoch 8/10] step 1700/13933 (7.0 it/s) [Epoch 8/10] step 1750/13933 (7.0 it/s) [Epoch 8/10] step 1800/13933 (7.0 it/s) [Epoch 8/10] step 1850/13933 (7.0 it/s) [Epoch 8/10] step 1900/13933 (7.0 it/s) [Epoch 8/10] step 1950/13933 (6.9 it/s) [Epoch 8/10] step 2000/13933 (6.9 it/s) [Epoch 8/10] step 2050/13933 (6.9 it/s) [Epoch 8/10] step 2100/13933 (6.9 it/s) [Epoch 8/10] step 2150/13933 (6.9 it/s) [Epoch 8/10] step 2200/13933 (6.8 it/s) [Epoch 8/10] step 2250/13933 (6.9 it/s) [Epoch 8/10] step 2300/13933 (6.8 it/s) [Epoch 8/10] step 2350/13933 (6.8 it/s) [Epoch 8/10] step 2400/13933 (6.8 it/s) [Epoch 8/10] step 2450/13933 (6.8 it/s) [Epoch 8/10] step 2500/13933 (6.8 it/s) [Epoch 8/10] step 2550/13933 (6.8 it/s) [Epoch 8/10] step 2600/13933 (6.8 it/s) [Epoch 8/10] step 2650/13933 (6.8 it/s) [Epoch 8/10] step 2700/13933 (6.8 it/s) [Epoch 8/10] step 2750/13933 (6.8 it/s) [Epoch 8/10] step 2800/13933 (6.8 it/s) [Epoch 8/10] step 2850/13933 (6.8 it/s) [Epoch 8/10] step 2900/13933 (6.8 it/s) [Epoch 8/10] step 2950/13933 (6.8 it/s) [Epoch 8/10] step 3000/13933 (6.8 it/s) [Epoch 8/10] step 3050/13933 (6.8 it/s) [Epoch 8/10] step 3100/13933 (6.8 it/s) [Epoch 8/10] step 3150/13933 (6.8 it/s) [Epoch 8/10] step 3200/13933 (6.8 it/s) [Epoch 8/10] step 3250/13933 (6.8 it/s) [Epoch 8/10] step 3300/13933 (6.8 it/s) [Epoch 8/10] step 3350/13933 (6.8 it/s) [Epoch 8/10] step 3400/13933 (6.8 it/s) [Epoch 8/10] step 3450/13933 (6.8 it/s) [Epoch 8/10] step 3500/13933 (6.8 it/s) [Epoch 8/10] step 3550/13933 (6.8 it/s) [Epoch 8/10] step 3600/13933 (6.8 it/s) [Epoch 8/10] step 3650/13933 (6.8 it/s) [Epoch 8/10] step 3700/13933 (6.8 it/s) [Epoch 8/10] step 3750/13933 (6.8 it/s) [Epoch 8/10] step 3800/13933 (6.8 it/s) [Epoch 8/10] step 3850/13933 (6.8 it/s) [Epoch 8/10] step 3900/13933 (6.8 it/s) [Epoch 8/10] step 3950/13933 (6.8 it/s) [Epoch 8/10] step 4000/13933 (6.8 it/s) [Epoch 8/10] step 4050/13933 (6.8 it/s) [Epoch 8/10] step 4100/13933 (6.8 it/s) [Epoch 8/10] step 4150/13933 (6.8 it/s) [Epoch 8/10] step 4200/13933 (6.8 it/s) [Epoch 8/10] step 4250/13933 (6.8 it/s) [Epoch 8/10] step 4300/13933 (6.8 it/s) [Epoch 8/10] step 4350/13933 (6.8 it/s) [Epoch 8/10] step 4400/13933 (6.8 it/s) [Epoch 8/10] step 4450/13933 (6.8 it/s) [Epoch 8/10] step 4500/13933 (6.8 it/s) [Epoch 8/10] step 4550/13933 (6.8 it/s) [Epoch 8/10] step 4600/13933 (6.8 it/s) [Epoch 8/10] step 4650/13933 (6.8 it/s) [Epoch 8/10] step 4700/13933 (6.8 it/s) [Epoch 8/10] step 4750/13933 (6.8 it/s) [Epoch 8/10] step 4800/13933 (6.8 it/s) [Epoch 8/10] step 4850/13933 (6.7 it/s) [Epoch 8/10] step 4900/13933 (6.7 it/s) [Epoch 8/10] step 4950/13933 (6.7 it/s) [Epoch 8/10] step 5000/13933 (6.7 it/s) [Epoch 8/10] step 5050/13933 (6.7 it/s) [Epoch 8/10] step 5100/13933 (6.7 it/s) [Epoch 8/10] step 5150/13933 (6.7 it/s) [Epoch 8/10] step 5200/13933 (6.7 it/s) [Epoch 8/10] step 5250/13933 (6.7 it/s) [Epoch 8/10] step 5300/13933 (6.7 it/s) [Epoch 8/10] step 5350/13933 (6.7 it/s) [Epoch 8/10] step 5400/13933 (6.7 it/s) [Epoch 8/10] step 5450/13933 (6.7 it/s) [Epoch 8/10] step 5500/13933 (6.7 it/s) [Epoch 8/10] step 5550/13933 (6.7 it/s) [Epoch 8/10] step 5600/13933 (6.6 it/s) [Epoch 8/10] step 5650/13933 (6.6 it/s) [Epoch 8/10] step 5700/13933 (6.6 it/s) [Epoch 8/10] step 5750/13933 (6.6 it/s) [Epoch 8/10] step 5800/13933 (6.6 it/s) [Epoch 8/10] step 5850/13933 (6.6 it/s) [Epoch 8/10] step 5900/13933 (6.6 it/s) [Epoch 8/10] step 5950/13933 (6.6 it/s) [Epoch 8/10] step 6000/13933 (6.6 it/s) [Epoch 8/10] step 6050/13933 (6.6 it/s) [Epoch 8/10] step 6100/13933 (6.6 it/s) [Epoch 8/10] step 6150/13933 (6.6 it/s) [Epoch 8/10] step 6200/13933 (6.6 it/s) [Epoch 8/10] step 6250/13933 (6.6 it/s) [Epoch 8/10] step 6300/13933 (6.6 it/s) [Epoch 8/10] step 6350/13933 (6.6 it/s) [Epoch 8/10] step 6400/13933 (6.6 it/s) [Epoch 8/10] step 6450/13933 (6.6 it/s) [Epoch 8/10] step 6500/13933 (6.6 it/s) [Epoch 8/10] step 6550/13933 (6.6 it/s) [Epoch 8/10] step 6600/13933 (6.5 it/s) [Epoch 8/10] step 6650/13933 (6.5 it/s) [Epoch 8/10] step 6700/13933 (6.6 it/s) [Epoch 8/10] step 6750/13933 (6.6 it/s) [Epoch 8/10] step 6800/13933 (6.6 it/s) [Epoch 8/10] step 6850/13933 (6.5 it/s) [Epoch 8/10] step 6900/13933 (6.5 it/s) [Epoch 8/10] step 6950/13933 (6.5 it/s) [Epoch 8/10] step 7000/13933 (6.5 it/s) [Epoch 8/10] step 7050/13933 (6.5 it/s) [Epoch 8/10] step 7100/13933 (6.5 it/s) [Epoch 8/10] step 7150/13933 (6.5 it/s) [Epoch 8/10] step 7200/13933 (6.5 it/s) [Epoch 8/10] step 7250/13933 (6.5 it/s) [Epoch 8/10] step 7300/13933 (6.5 it/s) [Epoch 8/10] step 7350/13933 (6.5 it/s) [Epoch 8/10] step 7400/13933 (6.5 it/s) [Epoch 8/10] step 7450/13933 (6.5 it/s) [Epoch 8/10] step 7500/13933 (6.5 it/s) [Epoch 8/10] step 7550/13933 (6.5 it/s) [Epoch 8/10] step 7600/13933 (6.5 it/s) [Epoch 8/10] step 7650/13933 (6.5 it/s) [Epoch 8/10] step 7700/13933 (6.5 it/s) [Epoch 8/10] step 7750/13933 (6.5 it/s) [Epoch 8/10] step 7800/13933 (6.5 it/s) [Epoch 8/10] step 7850/13933 (6.5 it/s) [Epoch 8/10] step 7900/13933 (6.5 it/s) [Epoch 8/10] step 7950/13933 (6.5 it/s) [Epoch 8/10] step 8000/13933 (6.5 it/s) [Epoch 8/10] step 8050/13933 (6.5 it/s) [Epoch 8/10] step 8100/13933 (6.5 it/s) [Epoch 8/10] step 8150/13933 (6.5 it/s) [Epoch 8/10] step 8200/13933 (6.5 it/s) [Epoch 8/10] step 8250/13933 (6.5 it/s) [Epoch 8/10] step 8300/13933 (6.5 it/s) [Epoch 8/10] step 8350/13933 (6.5 it/s) [Epoch 8/10] step 8400/13933 (6.5 it/s) [Epoch 8/10] step 8450/13933 (6.5 it/s) [Epoch 8/10] step 8500/13933 (6.5 it/s) [Epoch 8/10] step 8550/13933 (6.5 it/s) [Epoch 8/10] step 8600/13933 (6.5 it/s) [Epoch 8/10] step 8650/13933 (6.5 it/s) [Epoch 8/10] step 8700/13933 (6.5 it/s) [Epoch 8/10] step 8750/13933 (6.5 it/s) [Epoch 8/10] step 8800/13933 (6.5 it/s) [Epoch 8/10] step 8850/13933 (6.5 it/s) [Epoch 8/10] step 8900/13933 (6.5 it/s) [Epoch 8/10] step 8950/13933 (6.5 it/s) [Epoch 8/10] step 9000/13933 (6.5 it/s) [Epoch 8/10] step 9050/13933 (6.5 it/s) [Epoch 8/10] step 9100/13933 (6.5 it/s) [Epoch 8/10] step 9150/13933 (6.5 it/s) [Epoch 8/10] step 9200/13933 (6.5 it/s) [Epoch 8/10] step 9250/13933 (6.5 it/s) [Epoch 8/10] step 9300/13933 (6.5 it/s) [Epoch 8/10] step 9350/13933 (6.5 it/s) [Epoch 8/10] step 9400/13933 (6.5 it/s) [Epoch 8/10] step 9450/13933 (6.5 it/s) [Epoch 8/10] step 9500/13933 (6.5 it/s) [Epoch 8/10] step 9550/13933 (6.5 it/s) [Epoch 8/10] step 9600/13933 (6.5 it/s) [Epoch 8/10] step 9650/13933 (6.5 it/s) [Epoch 8/10] step 9700/13933 (6.5 it/s) [Epoch 8/10] step 9750/13933 (6.5 it/s) [Epoch 8/10] step 9800/13933 (6.5 it/s) [Epoch 8/10] step 9850/13933 (6.5 it/s) [Epoch 8/10] step 9900/13933 (6.5 it/s) [Epoch 8/10] step 9950/13933 (6.5 it/s) [Epoch 8/10] step 10000/13933 (6.5 it/s) [Epoch 8/10] step 10050/13933 (6.5 it/s) [Epoch 8/10] step 10100/13933 (6.5 it/s) [Epoch 8/10] step 10150/13933 (6.5 it/s) [Epoch 8/10] step 10200/13933 (6.5 it/s) [Epoch 8/10] step 10250/13933 (6.5 it/s) [Epoch 8/10] step 10300/13933 (6.5 it/s) [Epoch 8/10] step 10350/13933 (6.5 it/s) [Epoch 8/10] step 10400/13933 (6.5 it/s) [Epoch 8/10] step 10450/13933 (6.5 it/s) [Epoch 8/10] step 10500/13933 (6.5 it/s) [Epoch 8/10] step 10550/13933 (6.5 it/s) [Epoch 8/10] step 10600/13933 (6.5 it/s) [Epoch 8/10] step 10650/13933 (6.5 it/s) [Epoch 8/10] step 10700/13933 (6.5 it/s) [Epoch 8/10] step 10750/13933 (6.5 it/s) [Epoch 8/10] step 10800/13933 (6.5 it/s) [Epoch 8/10] step 10850/13933 (6.5 it/s) [Epoch 8/10] step 10900/13933 (6.5 it/s) [Epoch 8/10] step 10950/13933 (6.5 it/s) [Epoch 8/10] step 11000/13933 (6.5 it/s) [Epoch 8/10] step 11050/13933 (6.5 it/s) [Epoch 8/10] step 11100/13933 (6.5 it/s) [Epoch 8/10] step 11150/13933 (6.5 it/s) [Epoch 8/10] step 11200/13933 (6.5 it/s) [Epoch 8/10] step 11250/13933 (6.5 it/s) [Epoch 8/10] step 11300/13933 (6.5 it/s) [Epoch 8/10] step 11350/13933 (6.5 it/s) [Epoch 8/10] step 11400/13933 (6.5 it/s) [Epoch 8/10] step 11450/13933 (6.5 it/s) [Epoch 8/10] step 11500/13933 (6.5 it/s) [Epoch 8/10] step 11550/13933 (6.5 it/s) [Epoch 8/10] step 11600/13933 (6.5 it/s) [Epoch 8/10] step 11650/13933 (6.5 it/s) [Epoch 8/10] step 11700/13933 (6.5 it/s) [Epoch 8/10] step 11750/13933 (6.5 it/s) [Epoch 8/10] step 11800/13933 (6.5 it/s) [Epoch 8/10] step 11850/13933 (6.5 it/s) [Epoch 8/10] step 11900/13933 (6.5 it/s) [Epoch 8/10] step 11950/13933 (6.5 it/s) [Epoch 8/10] step 12000/13933 (6.5 it/s) [Epoch 8/10] step 12050/13933 (6.5 it/s) [Epoch 8/10] step 12100/13933 (6.5 it/s) [Epoch 8/10] step 12150/13933 (6.5 it/s) [Epoch 8/10] step 12200/13933 (6.5 it/s) [Epoch 8/10] step 12250/13933 (6.5 it/s) [Epoch 8/10] step 12300/13933 (6.5 it/s) [Epoch 8/10] step 12350/13933 (6.5 it/s) [Epoch 8/10] step 12400/13933 (6.5 it/s) [Epoch 8/10] step 12450/13933 (6.5 it/s) [Epoch 8/10] step 12500/13933 (6.5 it/s) [Epoch 8/10] step 12550/13933 (6.5 it/s) [Epoch 8/10] step 12600/13933 (6.5 it/s) [Epoch 8/10] step 12650/13933 (6.5 it/s) [Epoch 8/10] step 12700/13933 (6.5 it/s) [Epoch 8/10] step 12750/13933 (6.5 it/s) [Epoch 8/10] step 12800/13933 (6.5 it/s) [Epoch 8/10] step 12850/13933 (6.5 it/s) [Epoch 8/10] step 12900/13933 (6.5 it/s) [Epoch 8/10] step 12950/13933 (6.5 it/s) [Epoch 8/10] step 13000/13933 (6.5 it/s) [Epoch 8/10] step 13050/13933 (6.5 it/s) [Epoch 8/10] step 13100/13933 (6.5 it/s) [Epoch 8/10] step 13150/13933 (6.5 it/s) [Epoch 8/10] step 13200/13933 (6.5 it/s) [Epoch 8/10] step 13250/13933 (6.5 it/s) [Epoch 8/10] step 13300/13933 (6.5 it/s) [Epoch 8/10] step 13350/13933 (6.5 it/s) [Epoch 8/10] step 13400/13933 (6.5 it/s) [Epoch 8/10] step 13450/13933 (6.5 it/s) [Epoch 8/10] step 13500/13933 (6.5 it/s) [Epoch 8/10] step 13550/13933 (6.5 it/s) [Epoch 8/10] step 13600/13933 (6.5 it/s) [Epoch 8/10] step 13650/13933 (6.5 it/s) [Epoch 8/10] step 13700/13933 (6.5 it/s) [Epoch 8/10] step 13750/13933 (6.5 it/s) [Epoch 8/10] step 13800/13933 (6.5 it/s) [Epoch 8/10] step 13850/13933 (6.5 it/s) [Epoch 8/10] step 13900/13933 (6.5 it/s) 18:04:48 | INFO | trainer_inf~| \n+----------------------------+------------------------+ | Metric | Value | +----------------------------+------------------------+ | fit/loss | 0.08396278321743011 | | fit/pred_loss | 0.006814345717430115 | | fit/sigreg_loss | 0.859375 | | hardware/cpu_percent | 35.20000076293945 | | hardware/disk_read_mb_s | 1967.374755859375 | | hardware/disk_write_mb_s | 1.5341644287109375 | | hardware/net_recv_mb_s | 0.00013614434283226728 | | hardware/net_sent_mb_s | 0.00021928010392002761 | | hardware/ram_used_gb | 216.19174194335938 | | hardware/ram_used_pct | 42.900001525878906 | | hparams/lr_model_opt | 1.2482166766858427e-06 | | validate/loss | 0.12313934415578842 | | validate/loss_epoch | 0.12313934415578842 | | validate/pred_loss | 0.0032770915422588587 | | validate/pred_loss_epoch | 0.0032770915422588587 | | validate/sigreg_loss | 1.3318232297897339 | | validate/sigreg_loss_epoch | 1.3318232297897339 | +----------------------------+------------------------+ 18:04:49 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_009.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 18:04:49 | INFO | checkpoint_~| Checking for non PyTorch modules to save 18:04:49 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 18:04:49 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.49s write=0.14s rename=0.3ms) [Epoch 8/10] done in 2392.3s 18:04:49 | INFO | checkpoint_~| Checking for non PyTorch modules to save 18:04:49 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=8-step=125397.ckpt 18:04:50 | INFO | atomic_chec~| [atomic_save] ✓ epoch=8-step=125397.ckpt saved (206.8 MiB, serialise=0.41s write=0.14s rename=0.1ms) 18:04:50 | INFO | checkpoint_~| Checking for non PyTorch modules to save 18:04:50 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 18:04:50 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.48s write=0.14s rename=0.2ms) [Epoch 9/10] step 50/13933 (5.3 it/s) [Epoch 9/10] step 100/13933 (6.2 it/s) [Epoch 9/10] step 150/13933 (6.5 it/s) [Epoch 9/10] step 200/13933 (6.7 it/s) [Epoch 9/10] step 250/13933 (6.8 it/s) [Epoch 9/10] step 300/13933 (6.5 it/s) [Epoch 9/10] step 350/13933 (6.3 it/s) [Epoch 9/10] step 400/13933 (6.1 it/s) [Epoch 9/10] step 450/13933 (6.0 it/s) [Epoch 9/10] step 500/13933 (6.1 it/s) [Epoch 9/10] step 550/13933 (6.2 it/s) [Epoch 9/10] step 600/13933 (6.2 it/s) [Epoch 9/10] step 650/13933 (6.3 it/s) [Epoch 9/10] step 700/13933 (6.3 it/s) [Epoch 9/10] step 750/13933 (6.2 it/s) [Epoch 9/10] step 800/13933 (6.3 it/s) [Epoch 9/10] step 850/13933 (6.3 it/s) [Epoch 9/10] step 900/13933 (6.4 it/s) [Epoch 9/10] step 950/13933 (6.4 it/s) [Epoch 9/10] step 1000/13933 (6.3 it/s) [Epoch 9/10] step 1050/13933 (6.3 it/s) [Epoch 9/10] step 1100/13933 (6.4 it/s) [Epoch 9/10] step 1150/13933 (6.4 it/s) [Epoch 9/10] step 1200/13933 (6.4 it/s) [Epoch 9/10] step 1250/13933 (6.4 it/s) [Epoch 9/10] step 1300/13933 (6.4 it/s) [Epoch 9/10] step 1350/13933 (6.4 it/s) [Epoch 9/10] step 1400/13933 (6.4 it/s) [Epoch 9/10] step 1450/13933 (6.4 it/s) [Epoch 9/10] step 1500/13933 (6.4 it/s) [Epoch 9/10] step 1550/13933 (6.5 it/s) [Epoch 9/10] step 1600/13933 (6.4 it/s) [Epoch 9/10] step 1650/13933 (6.4 it/s) [Epoch 9/10] step 1700/13933 (6.4 it/s) [Epoch 9/10] step 1750/13933 (6.4 it/s) [Epoch 9/10] step 1800/13933 (6.5 it/s) [Epoch 9/10] step 1850/13933 (6.5 it/s) [Epoch 9/10] step 1900/13933 (6.5 it/s) [Epoch 9/10] step 1950/13933 (6.4 it/s) [Epoch 9/10] step 2000/13933 (6.4 it/s) [Epoch 9/10] step 2050/13933 (6.5 it/s) [Epoch 9/10] step 2100/13933 (6.5 it/s) [Epoch 9/10] step 2150/13933 (6.5 it/s) [Epoch 9/10] step 2200/13933 (6.5 it/s) [Epoch 9/10] step 2250/13933 (6.5 it/s) [Epoch 9/10] step 2300/13933 (6.5 it/s) [Epoch 9/10] step 2350/13933 (6.5 it/s) [Epoch 9/10] step 2400/13933 (6.5 it/s) [Epoch 9/10] step 2450/13933 (6.5 it/s) [Epoch 9/10] step 2500/13933 (6.5 it/s) [Epoch 9/10] step 2550/13933 (6.5 it/s) [Epoch 9/10] step 2600/13933 (6.5 it/s) [Epoch 9/10] step 2650/13933 (6.5 it/s) [Epoch 9/10] step 2700/13933 (6.5 it/s) [Epoch 9/10] step 2750/13933 (6.5 it/s) [Epoch 9/10] step 2800/13933 (6.5 it/s) [Epoch 9/10] step 2850/13933 (6.5 it/s) [Epoch 9/10] step 2900/13933 (6.5 it/s) [Epoch 9/10] step 2950/13933 (6.5 it/s) [Epoch 9/10] step 3000/13933 (6.5 it/s) [Epoch 9/10] step 3050/13933 (6.5 it/s) [Epoch 9/10] step 3100/13933 (6.5 it/s) [Epoch 9/10] step 3150/13933 (6.5 it/s) [Epoch 9/10] step 3200/13933 (6.5 it/s) [Epoch 9/10] step 3250/13933 (6.5 it/s) [Epoch 9/10] step 3300/13933 (6.5 it/s) [Epoch 9/10] step 3350/13933 (6.5 it/s) [Epoch 9/10] step 3400/13933 (6.5 it/s) [Epoch 9/10] step 3450/13933 (6.5 it/s) [Epoch 9/10] step 3500/13933 (6.5 it/s) [Epoch 9/10] step 3550/13933 (6.5 it/s) [Epoch 9/10] step 3600/13933 (6.6 it/s) [Epoch 9/10] step 3650/13933 (6.5 it/s) [Epoch 9/10] step 3700/13933 (6.5 it/s) [Epoch 9/10] step 3750/13933 (6.6 it/s) [Epoch 9/10] step 3800/13933 (6.6 it/s) [Epoch 9/10] step 3850/13933 (6.5 it/s) [Epoch 9/10] step 3900/13933 (6.6 it/s) [Epoch 9/10] step 3950/13933 (6.6 it/s) [Epoch 9/10] step 4000/13933 (6.6 it/s) [Epoch 9/10] step 4050/13933 (6.6 it/s) [Epoch 9/10] step 4100/13933 (6.5 it/s) [Epoch 9/10] step 4150/13933 (6.5 it/s) [Epoch 9/10] step 4200/13933 (6.5 it/s) [Epoch 9/10] step 4250/13933 (6.5 it/s) [Epoch 9/10] step 4300/13933 (6.5 it/s) [Epoch 9/10] step 4350/13933 (6.5 it/s) [Epoch 9/10] step 4400/13933 (6.5 it/s) [Epoch 9/10] step 4450/13933 (6.5 it/s) [Epoch 9/10] step 4500/13933 (6.5 it/s) [Epoch 9/10] step 4550/13933 (6.5 it/s) [Epoch 9/10] step 4600/13933 (6.5 it/s) [Epoch 9/10] step 4650/13933 (6.6 it/s) [Epoch 9/10] step 4700/13933 (6.5 it/s) [Epoch 9/10] step 4750/13933 (6.5 it/s) [Epoch 9/10] step 4800/13933 (6.6 it/s) [Epoch 9/10] step 4850/13933 (6.6 it/s) [Epoch 9/10] step 4900/13933 (6.6 it/s) [Epoch 9/10] step 4950/13933 (6.6 it/s) [Epoch 9/10] step 5000/13933 (6.5 it/s) [Epoch 9/10] step 5050/13933 (6.5 it/s) [Epoch 9/10] step 5100/13933 (6.5 it/s) [Epoch 9/10] step 5150/13933 (6.5 it/s) [Epoch 9/10] step 5200/13933 (6.5 it/s) [Epoch 9/10] step 5250/13933 (6.5 it/s) [Epoch 9/10] step 5300/13933 (6.5 it/s) [Epoch 9/10] step 5350/13933 (6.5 it/s) [Epoch 9/10] step 5400/13933 (6.5 it/s) [Epoch 9/10] step 5450/13933 (6.5 it/s) [Epoch 9/10] step 5500/13933 (6.5 it/s) [Epoch 9/10] step 5550/13933 (6.5 it/s) [Epoch 9/10] step 5600/13933 (6.5 it/s) [Epoch 9/10] step 5650/13933 (6.5 it/s) [Epoch 9/10] step 5700/13933 (6.5 it/s) [Epoch 9/10] step 5750/13933 (6.5 it/s) [Epoch 9/10] step 5800/13933 (6.5 it/s) [Epoch 9/10] step 5850/13933 (6.5 it/s) [Epoch 9/10] step 5900/13933 (6.5 it/s) [Epoch 9/10] step 5950/13933 (6.5 it/s) [Epoch 9/10] step 6000/13933 (6.5 it/s) [Epoch 9/10] step 6050/13933 (6.5 it/s) [Epoch 9/10] step 6100/13933 (6.5 it/s) [Epoch 9/10] step 6150/13933 (6.5 it/s) [Epoch 9/10] step 6200/13933 (6.5 it/s) [Epoch 9/10] step 6250/13933 (6.5 it/s) [Epoch 9/10] step 6300/13933 (6.5 it/s) [Epoch 9/10] step 6350/13933 (6.5 it/s) [Epoch 9/10] step 6400/13933 (6.5 it/s) [Epoch 9/10] step 6450/13933 (6.5 it/s) [Epoch 9/10] step 6500/13933 (6.5 it/s) [Epoch 9/10] step 6550/13933 (6.5 it/s) [Epoch 9/10] step 6600/13933 (6.5 it/s) [Epoch 9/10] step 6650/13933 (6.5 it/s) [Epoch 9/10] step 6700/13933 (6.5 it/s) [Epoch 9/10] step 6750/13933 (6.5 it/s) [Epoch 9/10] step 6800/13933 (6.5 it/s) [Epoch 9/10] step 6850/13933 (6.6 it/s) [Epoch 9/10] step 6900/13933 (6.6 it/s) [Epoch 9/10] step 6950/13933 (6.6 it/s) [Epoch 9/10] step 7000/13933 (6.6 it/s) [Epoch 9/10] step 7050/13933 (6.6 it/s) [Epoch 9/10] step 7100/13933 (6.6 it/s) [Epoch 9/10] step 7150/13933 (6.6 it/s) [Epoch 9/10] step 7200/13933 (6.6 it/s) [Epoch 9/10] step 7250/13933 (6.6 it/s) [Epoch 9/10] step 7300/13933 (6.6 it/s) [Epoch 9/10] step 7350/13933 (6.6 it/s) [Epoch 9/10] step 7400/13933 (6.6 it/s) [Epoch 9/10] step 7450/13933 (6.6 it/s) [Epoch 9/10] step 7500/13933 (6.6 it/s) [Epoch 9/10] step 7550/13933 (6.6 it/s) [Epoch 9/10] step 7600/13933 (6.6 it/s) [Epoch 9/10] step 7650/13933 (6.6 it/s) [Epoch 9/10] step 7700/13933 (6.6 it/s) [Epoch 9/10] step 7750/13933 (6.6 it/s) [Epoch 9/10] step 7800/13933 (6.6 it/s) [Epoch 9/10] step 7850/13933 (6.6 it/s) [Epoch 9/10] step 7900/13933 (6.6 it/s) [Epoch 9/10] step 7950/13933 (6.6 it/s) [Epoch 9/10] step 8000/13933 (6.6 it/s) [Epoch 9/10] step 8050/13933 (6.6 it/s) [Epoch 9/10] step 8100/13933 (6.6 it/s) [Epoch 9/10] step 8150/13933 (6.6 it/s) [Epoch 9/10] step 8200/13933 (6.6 it/s) [Epoch 9/10] step 8250/13933 (6.6 it/s) [Epoch 9/10] step 8300/13933 (6.6 it/s) [Epoch 9/10] step 8350/13933 (6.6 it/s) [Epoch 9/10] step 8400/13933 (6.6 it/s) [Epoch 9/10] step 8450/13933 (6.5 it/s) [Epoch 9/10] step 8500/13933 (6.5 it/s) [Epoch 9/10] step 8550/13933 (6.5 it/s) [Epoch 9/10] step 8600/13933 (6.5 it/s) [Epoch 9/10] step 8650/13933 (6.5 it/s) [Epoch 9/10] step 8700/13933 (6.6 it/s) [Epoch 9/10] step 8750/13933 (6.5 it/s) [Epoch 9/10] step 8800/13933 (6.5 it/s) [Epoch 9/10] step 8850/13933 (6.5 it/s) [Epoch 9/10] step 8900/13933 (6.5 it/s) [Epoch 9/10] step 8950/13933 (6.5 it/s) [Epoch 9/10] step 9000/13933 (6.5 it/s) [Epoch 9/10] step 9050/13933 (6.5 it/s) [Epoch 9/10] step 9100/13933 (6.5 it/s) [Epoch 9/10] step 9150/13933 (6.5 it/s) [Epoch 9/10] step 9200/13933 (6.5 it/s) [Epoch 9/10] step 9250/13933 (6.5 it/s) [Epoch 9/10] step 9300/13933 (6.5 it/s) [Epoch 9/10] step 9350/13933 (6.5 it/s) [Epoch 9/10] step 9400/13933 (6.5 it/s) [Epoch 9/10] step 9450/13933 (6.5 it/s) [Epoch 9/10] step 9500/13933 (6.5 it/s) [Epoch 9/10] step 9550/13933 (6.5 it/s) [Epoch 9/10] step 9600/13933 (6.5 it/s) [Epoch 9/10] step 9650/13933 (6.5 it/s) [Epoch 9/10] step 9700/13933 (6.5 it/s) [Epoch 9/10] step 9750/13933 (6.5 it/s) [Epoch 9/10] step 9800/13933 (6.5 it/s) [Epoch 9/10] step 9850/13933 (6.5 it/s) [Epoch 9/10] step 9900/13933 (6.5 it/s) [Epoch 9/10] step 9950/13933 (6.5 it/s) [Epoch 9/10] step 10000/13933 (6.5 it/s) [Epoch 9/10] step 10050/13933 (6.5 it/s) [Epoch 9/10] step 10100/13933 (6.5 it/s) [Epoch 9/10] step 10150/13933 (6.5 it/s) [Epoch 9/10] step 10200/13933 (6.5 it/s) [Epoch 9/10] step 10250/13933 (6.5 it/s) [Epoch 9/10] step 10300/13933 (6.5 it/s) [Epoch 9/10] step 10350/13933 (6.5 it/s) [Epoch 9/10] step 10400/13933 (6.5 it/s) [Epoch 9/10] step 10450/13933 (6.5 it/s) [Epoch 9/10] step 10500/13933 (6.5 it/s) [Epoch 9/10] step 10550/13933 (6.5 it/s) [Epoch 9/10] step 10600/13933 (6.5 it/s) [Epoch 9/10] step 10650/13933 (6.5 it/s) [Epoch 9/10] step 10700/13933 (6.5 it/s) [Epoch 9/10] step 10750/13933 (6.5 it/s) [Epoch 9/10] step 10800/13933 (6.5 it/s) [Epoch 9/10] step 10850/13933 (6.5 it/s) [Epoch 9/10] step 10900/13933 (6.5 it/s) [Epoch 9/10] step 10950/13933 (6.5 it/s) [Epoch 9/10] step 11000/13933 (6.5 it/s) [Epoch 9/10] step 11050/13933 (6.5 it/s) [Epoch 9/10] step 11100/13933 (6.5 it/s) [Epoch 9/10] step 11150/13933 (6.5 it/s) [Epoch 9/10] step 11200/13933 (6.5 it/s) [Epoch 9/10] step 11250/13933 (6.5 it/s) [Epoch 9/10] step 11300/13933 (6.5 it/s) [Epoch 9/10] step 11350/13933 (6.5 it/s) [Epoch 9/10] step 11400/13933 (6.5 it/s) [Epoch 9/10] step 11450/13933 (6.5 it/s) [Epoch 9/10] step 11500/13933 (6.5 it/s) [Epoch 9/10] step 11550/13933 (6.5 it/s) [Epoch 9/10] step 11600/13933 (6.5 it/s) [Epoch 9/10] step 11650/13933 (6.5 it/s) [Epoch 9/10] step 11700/13933 (6.5 it/s) [Epoch 9/10] step 11750/13933 (6.5 it/s) [Epoch 9/10] step 11800/13933 (6.5 it/s) [Epoch 9/10] step 11850/13933 (6.5 it/s) [Epoch 9/10] step 11900/13933 (6.5 it/s) [Epoch 9/10] step 11950/13933 (6.5 it/s) [Epoch 9/10] step 12000/13933 (6.5 it/s) [Epoch 9/10] step 12050/13933 (6.5 it/s) [Epoch 9/10] step 12100/13933 (6.5 it/s) [Epoch 9/10] step 12150/13933 (6.5 it/s) [Epoch 9/10] step 12200/13933 (6.5 it/s) [Epoch 9/10] step 12250/13933 (6.5 it/s) [Epoch 9/10] step 12300/13933 (6.5 it/s) [Epoch 9/10] step 12350/13933 (6.5 it/s) [Epoch 9/10] step 12400/13933 (6.5 it/s) [Epoch 9/10] step 12450/13933 (6.5 it/s) [Epoch 9/10] step 12500/13933 (6.5 it/s) [Epoch 9/10] step 12550/13933 (6.5 it/s) [Epoch 9/10] step 12600/13933 (6.5 it/s) [Epoch 9/10] step 12650/13933 (6.5 it/s) [Epoch 9/10] step 12700/13933 (6.5 it/s) [Epoch 9/10] step 12750/13933 (6.5 it/s) [Epoch 9/10] step 12800/13933 (6.5 it/s) [Epoch 9/10] step 12850/13933 (6.5 it/s) [Epoch 9/10] step 12900/13933 (6.5 it/s) [Epoch 9/10] step 12950/13933 (6.5 it/s) [Epoch 9/10] step 13000/13933 (6.5 it/s) [Epoch 9/10] step 13050/13933 (6.5 it/s) [Epoch 9/10] step 13100/13933 (6.5 it/s) [Epoch 9/10] step 13150/13933 (6.5 it/s) [Epoch 9/10] step 13200/13933 (6.5 it/s) [Epoch 9/10] step 13250/13933 (6.5 it/s) [Epoch 9/10] step 13300/13933 (6.5 it/s) [Epoch 9/10] step 13350/13933 (6.5 it/s) [Epoch 9/10] step 13400/13933 (6.5 it/s) [Epoch 9/10] step 13450/13933 (6.5 it/s) [Epoch 9/10] step 13500/13933 (6.5 it/s) [Epoch 9/10] step 13550/13933 (6.5 it/s) [Epoch 9/10] step 13600/13933 (6.5 it/s) [Epoch 9/10] step 13650/13933 (6.5 it/s) [Epoch 9/10] step 13700/13933 (6.5 it/s) [Epoch 9/10] step 13750/13933 (6.5 it/s) [Epoch 9/10] step 13800/13933 (6.5 it/s) [Epoch 9/10] step 13850/13933 (6.5 it/s) [Epoch 9/10] step 13900/13933 (6.5 it/s) 18:44:30 | INFO | trainer_inf~| \n+----------------------------+----------------------+ | Metric | Value | +----------------------------+----------------------+ | fit/loss | 0.0887174904346466 | | fit/pred_loss | 0.007662805262953043 | | fit/sigreg_loss | 0.8984375 | | hardware/cpu_percent | 20.299999237060547 | | hardware/disk_read_mb_s | 1608.10986328125 | | hardware/disk_write_mb_s | 0.3285861313343048 | | hardware/net_recv_mb_s | 0.0 | | hardware/net_sent_mb_s | 0.0 | | hardware/ram_used_gb | 201.94219970703125 | | hardware/ram_used_pct | 40.099998474121094 | | hparams/lr_model_opt | 0.0 | | validate/loss | 0.12302578240633011 | | validate/loss_epoch | 0.12302578240633011 | | validate/pred_loss | 0.003187905764207244 | | validate/pred_loss_epoch | 0.003187905764207244 | | validate/sigreg_loss | 1.3315798044204712 | | validate/sigreg_loss_epoch | 1.3315798044204712 | +----------------------------+----------------------+ 18:44:30 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_010.pt 📦📦📦 18:44:30 | INFO | utils.py | 📦📦📦 Model saved to /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/weights_epoch_010.pt 📦📦📦 `weights_only` was not set, defaulting to `False`. 18:44:30 | INFO | checkpoint_~| Checking for non PyTorch modules to save 18:44:30 | INFO | atomic_chec~| [atomic_save] → /root/.stable_worldmodel/checkpoints/pusht/2026-10-10_120411_img224_s3072/lewm_weights.ckpt 18:44:30 | INFO | atomic_chec~| [atomic_save] ✓ lewm_weights.ckpt saved (206.8 MiB, serialise=0.34s write=0.11s rename=2.6ms) [Epoch 9/10] done in 2380.1s 18:44:30 | INFO | checkpoint_~| Checking for non PyTorch modules to save 18:44:30 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/epoch=9-step=139330.ckpt 18:44:31 | INFO | atomic_chec~| [atomic_save] ✓ epoch=9-step=139330.ckpt saved (206.8 MiB, serialise=0.34s write=0.12s rename=0.1ms) 18:44:31 | INFO | checkpoint_~| Checking for non PyTorch modules to save 18:44:31 | INFO | atomic_chec~| [atomic_save] → /root/.cache/stable-pretraining/runs/20261010/120411/0a2e8732822c/checkpoints/last.ckpt 18:44:31 | INFO | atomic_chec~| [atomic_save] ✓ last.ckpt saved (206.8 MiB, serialise=0.35s write=0.13s rename=0.1ms) `Trainer.fit` stopped: `max_epochs=10` reached. Epoch 9/9 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13933/13933 0:35:38 • 0:00:00 6.75it/s 18:44:37 | INFO | utils.py | ── SignalHandlers (after Trainer.fit() returned) ── 18:44:37 | INFO | manager.py | SIGUSR1 → SIG_DFL (default OS action — terminate process) 18:44:37 | INFO | manager.py | SIGUSR2 → SIG_DFL (default OS action — terminate process) 18:44:37 | INFO | manager.py | SIGCONT → SIG_DFL (default OS action — terminate process) 18:44:37 | INFO | manager.py | SIGTERM → SIG_DFL (default OS action — terminate process)