--- name: lora-finetune description: | Use this skill to prepare training data and run LoRA fine-tuning of JoyAI-VL-Interaction-Preview into a custom proactive speak/silent specialist for a user-defined target event, on a machine already provisioned by `training-machine-provisioning`. Takes two user-supplied inputs — a local video file (or files) and a one-line description of the target event — and automates everything except the event-timestamp manifest, which stays manual. Use when data prep or fine-tuning invocation is explicitly requested compatibility: | Requires `training-machine-provisioning` to have already run against the target instance — SSH access, the `~/training/.venv` environment, and `~/training/base-model` (the cloned checkpoint) all need to already exist. metadata: category: training role: dependent --- Preparing Training Data ------------------------ Two things can only come from the user — a local video file (or files) and a one-line description of the target event to specialize for. Everything below through Step 5 is mechanical and should be run directly (by the agent, not handed to the user as manual steps) once those two inputs are known. If either is missing, ask for it before proceeding; don't guess a target-event description. The one exception is `manifest.jsonl` (Step 3 below): **that stays manual.** It records *when* the target event actually happens in the footage, which only the user (or whoever screened it) knows — auto-generating it would mean an unfine-tuned model labeling its own training data, defeating the point of fine-tuning it. Do not attempt to infer event timestamps automatically. Everything below runs **on the instance**, under the `ubuntu` user — `~` means `/home/ubuntu` there, not wherever this skill's own files live locally. 1. Copy this skill's `scripts/` directory onto the instance — `training-machine-provisioning` never put it there (that skill only ran `apt`/`uv`/`git`/stdlib commands): ```bash scp -i -r scripts ubuntu@:~/training/scripts ``` (run from the local machine that has this skill checked out — `scripts` here is this skill's own [`scripts/`](./scripts) directory; safe to re-run, it just overwrites) 2. Upload the video(s) and write the target-event prompt, both automated from the two user-supplied inputs: ```bash ssh -i ubuntu@ "mkdir -p ~/training/data/videos ~/training/config" scp -i

ubuntu@:~/training/data/videos/ ssh -i ubuntu@ "cat > ~/training/config/target_event.txt" <<< "" ``` Every other part of this pipeline is scenario-agnostic — `target_event.txt` is the only thing that changes between different target events, and it's reused verbatim at both training and inference time. See [references/dataset-and-framework.md](./references/dataset-and-framework.md) for why the two have to match. 3. **Manual step.** Write `~/training/data/manifest.jsonl` on the instance, referencing the filename(s) uploaded in Step 2, labeling where the target event occurs in each video. Full schema, class-balance guidance, and how much footage a Pilot vs. Moderate vs. Full-scale run needs are in [references/dataset-and-framework.md](./references/dataset-and-framework.md). 4. Once the manifest exists, build the labeled, frame-extracted LLaMA-Factory dataset — run this over SSH, from `~/training` on the instance so the `scripts/prepare_dataset.py` relative path resolves against the copy from Step 1: ```bash ssh -i ubuntu@ " cd ~/training && source .venv/bin/activate python scripts/prepare_dataset.py \ --manifest ~/training/data/manifest.jsonl \ --videos-dir ~/training/data/videos \ --target-event ~/training/config/target_event.txt \ --out ~/training/data/prepared \ --silence-ratio 3.0 " ``` This writes `~/training/data/prepared/dataset/joyai_target_event.json` (the ShareGPT-format examples) and `~/training/data/prepared/dataset/dataset_info.json` (registers it) — `train_config.yaml`'s `dataset_dir` points here. 5. Write the training config onto the instance, filling in `default_system` from the exact contents of `target_event.txt` written in Step 2 (must be byte-identical — see [references/dataset-and-framework.md](./references/dataset-and-framework.md)) and setting `output_dir` for this run. Use `scripts/train_config.yaml` (copied over in Step 1) as the base and rewrite it in full via heredoc, the same way `target_event.txt` itself was written — don't `sed`-substitute in place, the target-event text isn't guaranteed to be `sed`-delimiter-safe: ```bash ssh -i ubuntu@ "cat > ~/training/config/train_smoke.yaml" <<'EOF' EOF ``` Produce a second copy (`train_run1.yaml`, `output_dir: /home/ubuntu/training/adapters/run-1`, **no** `max_steps` line) for the full run in the next section — same substitution, no step cap. **`llamafactory-cli train config.yaml --max_steps 20` does not work as a CLI override — confirmed against a real run.** It fails with `ValueError: Some keys are not used by the HfArgumentParser: ['--max_steps', '20']`; trailing args aren't merged into the YAML the way plain `argparse` scripts do. Put `max_steps: 20` directly in the smoke-test YAML instead, as above — `llamafactory-cli train ` takes no other arguments. Registering the Base Model (One-Time) --------------------------------------- Before the first fine-tune on a fresh instance, register the base checkpoint as a versioned MLflow Model so it's visible in the MLflow UI's Models tab and loadable back via `mlflow.transformers.load_model(...)` — safe to skip if already registered (re-running just adds another version, harmless but unnecessary). **Prerequisite: `mlflow ui` must already be running and reachable at `http://127.0.0.1:5000` on the instance — this step does not start it.** Registering a *Model* (as opposed to just logging run metrics/params) uses MLflow's newer artifact-store codepath, which requires an `mlflow-artifacts://` URI resolved through an actual running server; pointing `MLFLOW_TRACKING_URI` directly at the SQLite file the way run-tracking elsewhere in this skill does fails with `the tracking URI must be a valid http or https URI` — confirmed against a real deploy. Check first, and start it if nothing's running: ```bash ssh -i ubuntu@ "ps aux | grep 'mlflow ui' | grep -v grep || echo none-running" ``` If that printed `none-running`, start it the same way [`mlflow-dashboard-setup`](../mlflow-dashboard-setup)'s private path does (safe default — localhost-only, survives the SSH session via `setsid`): ```bash ssh -i ubuntu@ " cd ~/training && source .venv/bin/activate setsid nohup mlflow ui --backend-store-uri sqlite:////home/ubuntu/training/mlflow.db \ --allowed-hosts '127.0.0.1:5000,localhost:5000,127.0.0.1,localhost' --port 5000 \ > ~/training/mlflow-ui.log 2>&1 < /dev/null & echo launched " ``` (If `mlflow-https-setup` was already applied, `mlflow ui` is already running with its own domain-scoped `--allowed-hosts` — that's fine too, as long as `127.0.0.1:5000` is included in it; see that skill's Step 3 if not.) Now register: ```bash ssh -i ubuntu@ " cd ~/training && source .venv/bin/activate export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 python scripts/register_model.py --mode base \ --model-path /home/ubuntu/training/base-model \ --registered-name joyai-vl-interaction-preview " ``` If the server's `--allowed-hosts` doesn't include `127.0.0.1:5000` (not just the bare `127.0.0.1`), this call 403s the same way external requests do without the domain-with-port entry — see `mlflow-dashboard-setup`'s known-issues for the general rule; it applies here too, just for the loopback address instead of the public one. This copies the full ~17GB checkpoint into MLflow's own artifact store (`~/training/mlartifacts`) — a real second copy, not a reference; see `references/dataset-and-framework.md` for why that's the accepted tradeoff. It's slow (large-file uploads through MLflow's client are not high-throughput even over loopback — budget on the order of an hour, not minutes, and let it run in the background rather than assuming a long-idle process has hung). Smoke Test Before Committing to a Full Run -------------------------------------------- Run a tiny slice through the real trainer before trusting it with hours of GPU time (still on the instance, from `~/training`): ```bash cd ~/training source .venv/bin/activate export MLFLOW_TRACKING_URI=sqlite:////home/ubuntu/training/mlflow.db export MLFLOW_EXPERIMENT_NAME= llamafactory-cli train config/train_smoke.yaml ``` Confirm all three before proceeding: loss is finite and trending down over the 20 steps, `nvidia-smi` shows memory usage with headroom (not near the 48 GB ceiling), and a checkpoint saves successfully to `output_dir`. If any of these fail, fix it here — don't diagnose it 40 minutes into a Full-scale run. See [references/hardware-and-cost.md](./references/hardware-and-cost.md) for what to do if it OOMs — note that doc's OOM thresholds predate this LLaMA-Factory migration and need re-confirming here, not assumed unchanged. Running the Full LoRA SFT --------------------------- ```bash cd ~/training source .venv/bin/activate export MLFLOW_TRACKING_URI=sqlite:////home/ubuntu/training/mlflow.db export MLFLOW_EXPERIMENT_NAME= nohup llamafactory-cli train config/train_run1.yaml > train.log 2>&1 & disown ``` Monitor with `tail -f train.log` and `nvidia-smi`, or watch loss/learning-rate curves update live in MLflow. From your **local** machine, tunnel the instance's MLflow UI port over SSH: ```bash ssh -i -L 5000:localhost:5000 ubuntu@ \ "cd ~/training && source .venv/bin/activate && mlflow ui --backend-store-uri sqlite:////home/ubuntu/training/mlflow.db --port 5000 --host 127.0.0.1" ``` **The `sqlite:////` prefix needs exactly four slashes for an absolute path — confirmed against a real deploy.** MLflow's SQLite URIs follow SQLAlchemy's convention: `sqlite:///relative/path` (three slashes) resolves *relative to the current working directory*, and silently creates a nested `/home/ubuntu/training/...` tree instead of erroring — it doesn't fail loudly, it just writes the database somewhere unintended. Absolute paths need the fourth slash. This also means `MLFLOW_TRACKING_URI` and every `--backend-store-uri` in this skill must use the same four-slash form, not three. Also confirmed against a real deploy: **plain `file:///...mlruns` no longer works** on current MLflow (3.x) — the filesystem tracking backend is in maintenance mode and `mlflow ui`/`llamafactory-cli train` raise `MlflowException: The filesystem tracking backend ... is in maintenance mode` unless you either set `MLFLOW_ALLOW_FILE_STORE=true` or, better, use a SQLite backend as above — SQLite is still just a local file (no separate database server to run), so it costs nothing extra over the file store while staying on MLflow's supported path. Leave that running and open `http://localhost:5000` in a browser. Because every run under a given `MLFLOW_TRACKING_URI` is recorded to the same backing store, this dashboard accumulates every fine-tune you've ever run on this instance — filter by `MLFLOW_EXPERIMENT_NAME` (one per target event) to compare hyperparams and loss curves across runs of the same event, e.g. `smoke-test` vs. `run-1`, without restarting anything. This tunnel dies with the SSH session above — fine for watching one run finish. For a dashboard that outlives a single session (reopen later, or reach it without an SSH client at all), see the [`mlflow-dashboard-setup`](../mlflow-dashboard-setup) skill. The result is a LoRA adapter directory (tens of MB, not a full model copy) under `output_dir` — load it on top of `~/training/base-model` via `peft`'s `PeftModel.from_pretrained` for evaluation, or use LLaMA-Factory's own `llamafactory-cli export` (merges the adapter into the base checkpoint) before handing weights to `inference-machine-provisioning` if you want it served as a plain checkpoint. ### Registering the Result Same prerequisite as "Registering the Base Model" above — `mlflow ui` must already be running and reachable at `http://127.0.0.1:5000`; check/start it the same way if this is a fresh session that skipped straight to the full run. Register the resulting adapter as a new version of this target event's own MLflow Model — since it's just the small adapter directory (no base weights involved), this is fast, unlike base-model registration above: ```bash ssh -i ubuntu@ " cd ~/training && source .venv/bin/activate export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 python scripts/register_model.py --mode adapter \ --base-model-path /home/ubuntu/training/base-model \ --adapter-path /home/ubuntu/training/adapters/run-1 \ --registered-name \ --experiment-name " ``` `--registered-name`/`--experiment-name` reuse the same target-event slug as `MLFLOW_EXPERIMENT_NAME` above — one registered Model per target event, versioned across every run against that event, each version linked back to the training run that produced it (the script looks up that run by querying the experiment for its most recent run, since `llamafactory-cli train`'s own MLflow integration starts and ends that run internally). To pick a registered version back up later — as the base for a further fine-tune on new data, or just for evaluation — `mlflow.transformers.load_model("models://", return_type="components")` returns the adapter's model+tokenizer+processor as a dict; combine with the base model the same way as above (`peft.PeftModel.from_pretrained(base_model, )`). **`return_type="components"` is required — confirmed against a real round-trip.** Without it, `load_model` tries to build a `Pipeline` and raises `MlflowException: This model has been saved with a processor. Processor objects are not compatible with Pipelines. Please load this model by specifying the 'return_type'='components'` — the same custom-processor issue as registration itself, just on the way back out. Also confirmed: loading the full base model this way pulls real weight bytes from MLflow's artifact store fresh each time (no local cache reuse observed across repeated calls) and briefly needs real memory headroom while materializing — a full round-trip peaked around 43 GB RSS on this checkpoint before settling back down once loaded, on a 61 GB-RAM instance with no swap configured. Comfortable here, but not something to run casually alongside an active training job. ⚠️ **Known issues / things that look like bugs but aren't** - `register_model.py`'s two modes use two different, both deliberately-chosen strategies against the same underlying problem — confirmed against real failures, not a style inconsistency: - **Base registration** passes the checkpoint directory to `mlflow.transformers.log_model` as a **path string**, not a loaded model. A live `{"model": ..., "processor": ...}` dict makes MLflow try to build an actual `transformers.pipeline(task=..., model=...)` internally before saving anything, and that fails on this checkpoint's custom multimodal processor with `Exception: Impossible to guess which processor to use` if the dict doesn't happen to contain exactly what the pipeline factory needs. The path-string form skips pipeline construction entirely (a plain directory copy), sidestepping the problem — but it requires a `config.json` in the given directory, which a bare PEFT adapter directory doesn't have. - **Adapter registration** can't use that shortcut (no `config.json`), so it goes through MLflow's PEFT path instead: a live `peft.PeftModel` plus `base_model_path` — which does mean loading the ~17GB base checkpoint into memory just to register the adapter. This revives the pipeline-guessing problem, and neither of `mlflow.transformers.log_model`'s dict-input forms can solve it here: the dict's allowed keys (`model`/`image_processor`/`tokenizer`/`feature_extractor`/`torch_dtype`) don't include a unified `processor`, and passing the processor's `tokenizer`+`image_processor` sub-components separately still fails the same guessing check — only `transformers.pipeline(task=..., model=..., processor=processor)` (the whole `Qwen3VLProcessor` object) actually works for this model+task. The fix: build that `Pipeline` object directly and pass the finished instance (not a dict) as `transformers_model` — `log_model` skips its own pipeline-construction attempt entirely when already handed a built `Pipeline`. - The vision tower is deliberately left untouched, same as before — but now declaratively, via LLaMA-Factory's `freeze_vision_tower: true` and `freeze_multi_modal_projector: true` defaults, rather than by `lora_target` accidentally not name-matching the vision tower's projection layers. `train_config.yaml` sets `lora_target: all`, which is safe specifically because those freeze defaults are on — don't set either freeze flag to `false` without re-checking trainable-parameter count for an order-of-magnitude jump. - Silence is the overwhelming majority class in raw 1fps-labeled footage. `prepare_dataset.py`'s `--silence-ratio` downsampling is not optional scaffolding — training on the raw class ratio produces a model that has learned to always predict silence at low loss. If a smoke-test run shows the loss hit near-zero suspiciously fast, check the silence ratio before trusting it. - Do not reach for a multi-GPU instance to speed up a slow Full-scale run — see the "multi-GPU cost trap" section in [references/hardware-and-cost.md](./references/hardware-and-cost.md) for why it doesn't help here and what the actual bottleneck usually is. That reasoning (video decode is CPU-bound, LoRA's trainable-parameter count is small) is framework-agnostic and still applies under LLaMA-Factory. - `prepare_dataset.py --max-frames` (default `4`) was the dominant lever on training VRAM under the previous custom-Trainer pipeline (5 frames reliably OOM'd, 4 ran clean at ~42.6/46 GiB) — re-confirm this holds under LLaMA-Factory's own batching/collation before trusting the old threshold; its per-device batching may have different peak-memory behavior than the old fixed-batch-size-1 collator. Re-run the smoke test with `nvidia-smi` open if you change `--max-frames` from the default. - `train_config.yaml` keeps `per_device_train_batch_size: 1`, carried over from the previous pipeline where it was a hard requirement (its custom collator couldn't batch examples with a variable frame count). LLaMA-Factory's own multimodal collator may handle this correctly at a batch size above `1`, which could reduce wall-clock — but that hasn't been validated for this checkpoint/dataset shape. Treat raising it as an experiment to smoke-test (watch `nvidia-smi` and confirm loss still looks sane), not a safe default change. - `default_system` in the training config must stay byte-identical to `~/training/config/target_event.txt` and to whatever prompt is used at inference time — see [references/dataset-and-framework.md](./references/dataset-and-framework.md), "The one prompt", for why a drift here isn't a bug in the model, it's a training/eval condition mismatch.

OPENML APERTURE