MOSS-FET

Since support for MOSS-Image in OneTrainer—the tool I usually use—is lagging, I’m testing a full fine-tuning (FT) process using a different tool.

I’ve set this up as a "Finetune Experimental Test" using a separate model instance.

ver. 0.1

Honestly, it’s been quite a struggle; I tried modifying various other tools, but they all failed—resulting in nothing but bizarre, garbled images. This time, I used "Fizgig." It’s an interesting tool, though it breaks the training process into four separate stages, which makes getting results a rather slow affair. It keeps the model in VRAM using NF4 quantization for the actual training, then writes the data back to BF16 format in the main system memory. Since the optimizer state is maintained in BF16 precision, the overall precision isn't actually that low; the theory is that any shortfall in computational precision is compensated for by a higher number of training steps (Since settings vary depending on the environment, it is highly likely that processing is being performed using BF16 in this instance.).

The output here is from the very first epoch, so this is strictly an experimental, preliminary test.