is there any example for an SFT / instruct finetune of Lizzy 7B on a regular
NVIDIA GPU with hugging face.
I found the finetuning samples for flower infra. But before digging into this I’d like to check the general capability of Lizzy7B compared to Mistral, Aperture, and Tilde with my domain data
Thanks for reaching out and for trying to use our latest model. There shouldn’t be anything fundamentally different about fine-tuning flwrlabs/Lizzy-7B versus other models of similar size. We’ve been fine-tuning the same checkpoint internally over the past few weeks and haven’t seen any model-specific issues that would explain the validation loss staying flat on its own.
If training loss goes down while validation loss does not, the first things I’d check are the usual ones: train/eval distribution mismatch, formatting differences between train and eval examples, sequence truncation effects, and hyperparameters that are slightly too aggressive for the dataset size. Since you’re already using a workflow that works on other models, I’d especially look closely at prompt formatting/chat template handling and whether the German instruct data is distributed similarly between train and eval.
If you can share a bit more detail about the data format and the exact training recipe, it may be easier to narrow down what’s causing the validation behavior.