一步一步用unsloth LoRA微调Qwen3

uv pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl==0.15.2 triton cut_cross_entropy unsloth_zoo
uv pip install sentencepiece protobuf datasets huggingface_hub hf_transfer
uv pip install --no-deps unsloth

from unsloth import FastLanguageModel
import torch

MODEL = "unsloth/Qwen3-14B"
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name =MODEL,
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
    full_finetuning = False
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 32,         
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 32,
    lora_dropout = 0,
    bias = "none",   
    use_gradient_checkpointing = "unsloth",
    use_rslora = False,  
    loftq_config = None,
)

# 获取高效微调模型 (PEFT model)
model = FastLanguageModel.get_peft_model(
    model,
    # 目标模块 (需进行 LoRA 适配的层)
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],
    # 使用梯度检查点 (优化显存技术)，这里指定 Unsloth 的实现
    use_gradient_checkpointing = "unsloth",
    r = 16,            # 秩 (Rank)，低秩适应层的大小
    lora_alpha = 4,    # LoRA 的缩放因子
    lora_dropout = 0,  # LoRA 层的 Dropout 概率 (防止过拟合)
    bias = "none"      # 是否微调偏置 (bias) 项
)

from datasets import load_dataset
reasoning_dataset = load_dataset("unsloth/OpenMathReasoning-mini", split = "cot")
non_reasoning_dataset = load_dataset("mlabonne/FineTome-100k", split = "train")

# 处理推理数据集
from pprint import pprint
pprint(reasoning_dataset[0])

# 手写一个 generate_conversation 函数，把原始的 problem（问题）和 generated_solution（生成的解答）强行打包成了标准的字典列表 [{"role": "user", ...}, {"role": "assistant", ...}]。
# 样例特征与翻译要求的呼应：
#    思维链 (Chain-of-Thought)： 一开始就是 <think> 标签，里面装满了诸如 "Okay, let's see...", "Let me try that..." 这样的碎碎念。这就是在教模型“三思而后行”，不要急于给出答案，把中间的推理步骤显式地打印出来。
#    清晰标记最终答案 (Mark the final answer clearly)： 在回复的最后，使用了 \boxed{14}。
#    兼容评估基准 (Compatibility with evaluation benchmarks)： 为什么要用 \boxed{}？因为主流的开源数学能力测试集（如 GSM8K, MATH），它们的自动判分脚本都是靠正则表达式去抓取 \boxed{} 里面的内容来判断对错的。如果你不教模型输出这个框，就算它算对了，在测试跑分时也会得零分。    
def generate_conversation(examples):
    problems  = examples["problem"]
    solutions = examples["generated_solution"]
    conversations = []
    for problem, solution in zip(problems, solutions):
        conversations.append([
            {"role" : "user",      "content" : problem},
            {"role" : "assistant", "content" : solution},
        ])
    return { "conversations": conversations, }

reasoning_conversations = tokenizer.apply_chat_template(
    reasoning_dataset.map(generate_conversation, batched = True)["conversations"],
    tokenize = False,
)

pprint(reasoning_dataset[0])

# 整理完的推理型的例子
<|im_start|>user
Given $\sqrt{x^2+165}-\sqrt{x^2-52}=7$ and $x$ is positive, find all possible values of $x$.<|im_end|>
<|im_start|>assistant
<think>
Okay, let's see. I need to solve the equation √(x² + 165) - √(x² - 52) = 7, and find all positive values of x. Hmm, radicals can be tricky, but maybe if I can eliminate the square roots by squaring both sides. Let me try that.

First, let me write down the equation again to make sure I have it right:

√(x² + 165) - √(x² - 52) = 7.

Okay, so the idea is to isolate one of the radicals and then square both sides. Let me try moving the second radical to the other side:

√(x² + 165) = 7 + √(x² - 52).

Now, if I square both sides, maybe I can get rid of the square roots. Let's do that:

(√(x² + 165))² = (7 + √(x² - 52))².

Simplifying the left side:

x² + 165 = 49 + 14√(x² - 52) + (√(x² - 52))².

The right side is expanded using the formula (a + b)² = a² + 2ab + b². So the right side becomes 7² + 2*7*√(x² - 52) + (√(x² - 52))², which is 49 + 14√(x² - 52) + (x² - 52).

So putting it all together:

x² + 165 = 49 + 14√(x² - 52) + x² - 52.

Hmm, let's simplify the right side. The x² terms will cancel out, right? Let's subtract x² from both sides:

165 = 49 + 14√(x² - 52) - 52.

Simplify the constants on the right:

49 - 52 is -3, so:

165 = -3 + 14√(x² - 52).

Now, add 3 to both sides to isolate the radical term:

165 + 3 = 14√(x² - 52).

So 168 = 14√(x² - 52).

Divide both sides by 14:

168 / 14 = √(x² - 52).

12 = √(x² - 52).

Now, square both sides again to eliminate the square root:

12² = x² - 52.

144 = x² - 52.

Add 52 to both sides:

144 + 52 = x².

196 = x².

So x = √196 = 14.

But wait, since the problem states that x is positive, we only take the positive root. So x = 14.

But hold on, when dealing with squaring equations, sometimes extraneous solutions can come up. I should check if this solution actually satisfies the original equation.

Let's plug x = 14 back into the original equation:

√(14² + 165) - √(14² - 52) = ?

Calculate each term:

14² is 196.

So first radical: √(196 + 165) = √361 = 19.

Second radical: √(196 - 52) = √144 = 12.

So 19 - 12 = 7, which is exactly the right-hand side. So yes, it checks out.

Therefore, the only solution is x = 14. Since the problem says x is positive, we don't have to consider negative roots. So I think that's the answer.
</think>

To solve the equation \(\sqrt{x^2 + 165} - \sqrt{x^2 - 52} = 7\) for positive \(x\), we proceed as follows:

1. Start with the given equation:
   \[
   \sqrt{x^2 + 165} - \sqrt{x^2 - 52} = 7
   \]

2. Isolate one of the square roots by moving \(\sqrt{x^2 - 52}\) to the right side:
   \[
   \sqrt{x^2 + 165} = 7 + \sqrt{x^2 - 52}
   \]

3. Square both sides to eliminate the square root on the left:
   \[
   (\sqrt{x^2 + 165})^2 = (7 + \sqrt{x^2 - 52})^2
   \]
   Simplifying both sides, we get:
   \[
   x^2 + 165 = 49 + 14\sqrt{x^2 - 52} + (x^2 - 52)
   \]

4. Combine like terms on the right side:
   \[
   x^2 + 165 = x^2 - 52 + 49 + 14\sqrt{x^2 - 52}
   \]
   Simplifying further:
   \[
   x^2 + 165 = x^2 - 3 + 14\sqrt{x^2 - 52}
   \]

5. Subtract \(x^2\) from both sides:
   \[
   165 = -3 + 14\sqrt{x^2 - 52}
   \]

6. Add 3 to both sides to isolate the term with the square root:
   \[
   168 = 14\sqrt{x^2 - 52}
   \]

7. Divide both sides by 14:
   \[
   12 = \sqrt{x^2 - 52}
   \]

8. Square both sides again to eliminate the square root:
   \[
   12^2 = x^2 - 52
   \]
   Simplifying:
   \[
   144 = x^2 - 52
   \]

9. Add 52 to both sides to solve for \(x^2\):
   \[
   196 = x^2
   \]

10. Take the positive square root (since \(x\) is positive):
    \[
    x = \sqrt{196} = 14
    \]

11. Verify the solution by substituting \(x = 14\) back into the original equation:
    \[
    \sqrt{14^2 + 165} - \sqrt{14^2 - 52} = \sqrt{196 + 165} - \sqrt{196 - 52} = \sqrt{361} - \sqrt{144} = 19 - 12 = 7
    \]
    The solution checks out.

Thus, the only positive solution is:
\[
\boxed{14}
\]<|im_end|>

# 非推理型
pprint(non_reasoning_dataset[0])

# 直接调用了开源微调神器 unsloth 自带的魔法库：standardize_sharegpt 和 apply_chat_template。
# ShareGPT 是目前开源界最流行的人机对话数据格式。
# 这两行代码会自动给数据穿上模型认识的“制服”（比如自动添加 <|im_start|> 和 <|im_end|> 这种特殊的控制字符）。
# 非推理型通常处理常识问答、代码编写、文本翻译等“知道就知道，不知道就不知道”的任务
from unsloth.chat_templates import standardize_sharegpt
dataset = standardize_sharegpt(non_reasoning_dataset)

non_reasoning_conversations = tokenizer.apply_chat_template(
    dataset["conversations"],
    tokenize = False,
)

pprint(non_reasoning_dataset[0])

# 整理完的非推理型的例子
<|im_start|>user
Explain what boolean operators are, what they do, and provide examples of how they can be used in programming. Additionally, describe the concept of operator precedence and provide examples of how it affects the evaluation of boolean expressions. Discuss the difference between short-circuit evaluation and normal evaluation in boolean expressions and demonstrate their usage in code. 

Furthermore, add the requirement that the code must be written in a language that does not support short-circuit evaluation natively, forcing the test taker to implement their own logic for short-circuit evaluation.

Finally, delve into the concept of truthiness and falsiness in programming languages, explaining how it affects the evaluation of boolean expressions. Add the constraint that the test taker must write code that handles cases where truthiness and falsiness are implemented differently across different programming languages.<|im_end|>
<|im_start|>assistant
<think>

</think>

Boolean operators are logical operators used in programming to manipulate boolean values. They operate on one or more boolean operands and return a boolean result. The three main boolean operators are "AND" (&&), "OR" (||), and "NOT" (!).

The "AND" operator returns true if both of its operands are true, and false otherwise. For example:

```python
x = 5
y = 10
result = (x > 0) and (y < 20)  # This expression evaluates to True
```

The "OR" operator returns true if at least one of its operands is true, and false otherwise. For example:

```python
x = 5
y = 10
result = (x > 0) or (y < 20)  # This expression evaluates to True
```

The "NOT" operator negates the boolean value of its operand. It returns true if the operand is false, and false if the operand is true. For example:

```python
x = 5
result = not (x > 10)  # This expression evaluates to True
```

Operator precedence refers to the order in which operators are evaluated in an expression. It ensures that expressions are evaluated correctly. In most programming languages, logical AND has higher precedence than logical OR. For example:

```python
result = True or False and False  # This expression is evaluated as (True or (False and False)), which is True
```

Short-circuit evaluation is a behavior where the second operand of a logical operator is not evaluated if the result can be determined based on the value of the first operand. In short-circuit evaluation, if the first operand of an "AND" operator is false, the second operand is not evaluated because the result will always be false. Similarly, if the first operand of an "OR" operator is true, the second operand is not evaluated because the result will always be true.

In programming languages that support short-circuit evaluation natively, you can use it to improve performance or avoid errors. For example:

```python
if x != 0 and (y / x) > 10:
    # Perform some operation
```

In languages without native short-circuit evaluation, you can implement your own logic to achieve the same behavior. Here's an example in pseudocode:

```
if x != 0 {
    if (y / x) > 10 {
        // Perform some operation
    }
}
```

Truthiness and falsiness refer to how non-boolean values are evaluated in boolean contexts. In many programming languages, non-zero numbers and non-empty strings are considered truthy, while zero, empty strings, and null/None values are considered falsy.

When evaluating boolean expressions, truthiness and falsiness come into play. For example:

```python
x = 5
result = x  # The value of x is truthy, so result is also truthy
```

To handle cases where truthiness and falsiness are implemented differently across programming languages, you can explicitly check the desired condition. For example:

```python
x = 5
result = bool(x)  # Explicitly converting x to a boolean value
```

This ensures that the result is always a boolean value, regardless of the language's truthiness and falsiness rules.<|im_end|>

print(len(reasoning_conversations))
19252

print(len(non_reasoning_conversations))
100000

data = pd.concat([
    pd.Series(reasoning_conversations),
    pd.Series(non_reasoning_subset)
])
data.name = "text"

from datasets import Dataset
combined_dataset = Dataset.from_pandas(pd.DataFrame(data))
combined_dataset = combined_dataset.shuffle(seed = 3407)

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = combined_dataset,
    eval_dataset = None,
    args = SFTConfig(
        dataset_text_field = "text",
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        report_to = "none",
    )
)

trainer_stats = trainer.train()


# 测测看
messages = [
    {"role" : "user", "content" : "Solve (x^2 + 5x + 6 = 0)."}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize = False,
    add_generation_prompt = True,
    enable_thinking = False,
)

from transformers import TextStreamer
_ = model.generate(
    **tokenizer(text, return_tensors = "pt").to("cuda"),
    max_new_tokens = 2048,
    temperature = 0.7, top_p = 0.8, top_k = 20,
    streamer = TextStreamer(tokenizer, skip_prompt = True),
)

# 保存
model.save_pretrained("qwen3_lora_model")
tokenizer.save_pretrained("qwen3_lora_model")

('qwen3_lora_model/tokenizer_config.json',
 'qwen3_lora_model/special_tokens_map.json',
 'qwen3_lora_model/vocab.json',
 'qwen3_lora_model/merges.txt',
 'qwen3_lora_model/added_tokens.json',
 'qwen3_lora_model/tokenizer.json')

from unsloth import FastLanguageModel
finetuned_model, finetuned_tokenizer = FastLanguageModel.from_pretrained(
        model_name = "qwen3_lora_model",
        max_seq_length = 2048,
        load_in_4bit = True,
    )
    

# 测一把
messages = [{"role" : "user", "content" : "Solve (x^2 + 5x + 6 = 0)."}]

text = finetuned_tokenizer.apply_chat_template(
    messages,
    tokenize = False,
    add_generation_prompt = True,
    enable_thinking = False,
)

from transformers import TextStreamer
_ = finetuned_model.generate(
    **finetuned_tokenizer(text, return_tensors = "pt").to("cuda"),
    max_new_tokens = 2048,
    temperature = 0.7, top_p = 0.8, top_k = 20,
    streamer = TextStreamer(finetuned_tokenizer, skip_prompt = True),
)