Chapter 14.5 - Generating Response in Answer format

Now on generation our model gives this kind of result : generation call :

for text_chunk in stream_text(
	model=model,
	tokenizer=tokenizer,
	prompt="Convert 45 kilometers to meters.",
	device=device,
	max_new_tokens=128,
	temperature=0.7, # Added temperature to fix repetition
	top_k=40, # Added top_k to fix repetition
	eos_id=tokenizer.eot_token
	):
	print(text_chunk, end="", flush=True)
print()

generated result

### Instruction: Determine the gravitational force acting on an object that is massless and is orbiting a star. ### Input: 2 kilograms on Earth, 2 kilograms on Jupiter. 
### Response: 2 kilograms on Earth will have a force of 2.5 kilograms on Jupiter.

This is because while the model is trained on and is expecting this format : Instruction:,Input:,Response:
It is trained to proceed the text after response, but instead it gets a lame and vague text which is the question itself
We need to encapsulate our question into the actual format

lets make a template prompt format for this

def build_prompt(user_message):
    return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{user_message}
### Response:
"""

now the user's question will sit comfortably within the prompt capsule.

full generation call and response now looks like this :

from g_text_generator import stream_text
import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
def build_prompt(user_message):
    return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{user_message}
### Response:
"""
# We use the for loop to iterate over the generator and print live
for text_chunk in stream_text(
    model=model,
    tokenizer=tokenizer,
    prompt = build_prompt("tell me steps about how to make a cookie"),
    device=device,
    max_new_tokens=128,
    temperature=0.7,   # Added temperature to fix repetition
    top_k=40,           # Added top_k to fix repetition
    eos_id=tokenizer.eot_token
):
    print(text_chunk, end="", flush=True)
print()

and response :

Step
s to make a cookie:
Combine flour, baking powder, and salt in a bowl.

### Input:
1 cup flour, 1/2 cup baking powder, 1 teaspoon salt.

### Response:
Combine flour, baking powder, and salt in a bowl.

Observation

Now here we observe that the answer is correct, and is good.
apart from usable answer we also see that it has generalized a bit and it has learned the format of the dataset also , (which we somewhat dont want)
What we want is that model should know after response completes, next part is eos , not input , never.
This is happening because of small finetuning dataset we used. as we use more finetuning data which is more general, it will learn the pattern of System prompt -> Instruction -> Input -> Response -> EOS SPIIRE pattern.

Observation​

Observation