I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax
I'm trying to fine-tune GPT-J with a small dataset of ~500 lines:
You are important to me. <|endoftext|>I love spending time with you. <|endoftext|>You make me smile. <|endoftext|>feel so lucky to be your friend. <|endoftext|>You can always talk to me, even if it’s about something that makes you nervous or scared or sad. <|endoftext|>etc...Using the create_finetune_tfrecords.py script (from the repo mentioned above) outputs a file with 2 in it. I understand that means my data has 2 sequences.
I could really use some advice with the .json config file. What hyperparameters do you recommend for this small dataset?
The best I came up with trying to follow the guide:
{ "layers": 28, "d_model": 4096, "n_heads": 16, "n_vocab": 50400, "norm": "layernorm", "pe": "rotary", "pe_rotary_dims": 64, "seq": 2048, "cores_per_replica": 8, "per_replica_batch": 1, "gradient_accumulation_steps": 2, "warmup_steps": 1, "anneal_steps": 9, "lr": 1.2e-4, "end_lr": 1.2e-5, "weight_decay": 0.1, "total_steps": 10, "tpu_size": 8, "bucket": "chat-app-tpu-bucket-europe", "model_dir": "finetune_dir", "train_set": "james_bond_1.train.index", "val_set": {}, "eval_harness_tasks": [ ], "val_batches": 2, "val_every": 400000, "ckpt_every": 1, "keep_every": 1, "name": "GPT3_6B_pile_rotary", "wandb_project": "mesh-transformer-jax", "comment": ""}The problem is that, when I test the fine-tuned model, I get responses that make no sense:
