I am working on the Transformer example demonstrated on TensorFlow's website. https://www.tensorflow.org/text/tutorials/transformer

In this example, Machine Translation model is trained to translate from Portuguese to English. The transformer is coded from scratch and other popular libraries like huggingface are not used.

Let's say I have another dataset which includes pairs of sentences of Portuguese and Finnish and let's say this dataset is fairly small. Since it is a small dataset, I want to use my model trained on Portuguese to English as a PreTrained model for creating the translation model for Portuguese to Finnish.

My question is, what are the key points to consider when using such a PreTrained model and changing ONLY its decoder output structure?