Hugging Face: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated
I’ve been trying out some of the Hugging Face tutorials and came across an interesting warning message while playing around with the google/flan-t5-large model. In this blog post, we’ll learn how to get rid of that warning.
I was running a variation of the getting started example:
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-large")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-large")
input_text = "Who is the UK Prime Minister? Explain step by step"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))The output from running this fragment of code is shown below:
<pad>The Prime Minister of the United Kingdom is Theresa May. Theresa May is theTo be fair, we have had a lot of Prime Ministers over the last few years, but it hasn’t been Theresa May for several years!
In any case, I also got the following error message when it executed the outputs = line:
/Users/markhneedham/projects/docs-bot/env/lib/python3.11/site-packages/transformers/generation/utils.py:1353: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
warnings.warn(I’m not actually setting a maximum length, but we seem to end up down this code path if no maximum is set.
You can also see that the output is chopped off.
We can fix that by setting the parameter max_new_tokens:
outputs = model.generate(input_ids, max_new_tokens=1000)
print(tokenizer.decode(outputs[0]))The new output is shown below:
<pad> The Prime Minister of the United Kingdom is Theresa May. Theresa May is the wife of David Cameron. David Cameron is the Prime Minister of the United Kingdom. So, the final answer is Theresa May.</s>The accuracy is pretty bad, but at least it’s not chopping off the output anymore.
About the author
I'm currently working on short form content at ClickHouse. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.