<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[What? Notes! - Medium]]></title>
        <description><![CDATA[Ironcat&#39;s notes for learning Linux/Web/Python tech - Medium]]></description>
        <link>https://medium.com/what-notes?source=rss----36654d023ed6---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>What? Notes! - Medium</title>
            <link>https://medium.com/what-notes?source=rss----36654d023ed6---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:57:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/what-notes" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Dump error log while calling smp_proccessor_id()]]></title>
            <link>https://medium.com/what-notes/dump-error-log-while-calling-smp-proccessor-id-489f37ddf421?source=rss----36654d023ed6---4</link>
            <guid isPermaLink="false">https://medium.com/p/489f37ddf421</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Ting Luo]]></dc:creator>
            <pubDate>Mon, 25 Jun 2018 15:45:45 GMT</pubDate>
            <atom:updated>2018-06-25T15:45:44.964Z</atom:updated>
            <content:encoded><![CDATA[<p>今天遇到一題smp_processor_id() 相關的問題, 故事是 driver 因為某些特殊需求而跟hotplug notifier 註冊callback. 當 cpu on/off 時hotplug notifier 通知我們的driver, 而 driver call smp_send_reschedule() 嘗試喚醒所有cpu時吐了一大串dumpstack…</p><p>driver call smp_send_reschedule(cpu) , 讓CPU 利用IPI 的方式被喚醒 (ex: cpu 1 send IPI to cpu 2 藉此喚醒cpu2). 接著call smp_cross_call(), 這邊smp_cross_call() 帶入 IPI_RESCHEDULE, 因此實際上會呼叫跟smp_croll_call註冊此 ipi num 的function</p><pre>void <a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_send_reschedule"><strong>smp_send_reschedule</strong></a>(int cpu)<br>{<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_cross_call"><strong>smp_cross_call</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/cpumask_of"><strong>cpumask_of</strong></a>(cpu), <a href="https://elixir.bootlin.com/linux/v4.8/ident/IPI_RESCHEDULE"><strong>IPI_RESCHEDULE</strong></a>);<br>}</pre><pre>static void <a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_cross_call"><strong>smp_cross_call</strong></a>(const struct <a href="https://elixir.bootlin.com/linux/v4.8/ident/cpumask"><strong>cpumask</strong></a> *<a href="https://elixir.bootlin.com/linux/v4.8/ident/target"><strong>target</strong></a>, unsigned int ipinr)<br>{<br>	trace_ipi_raise(<a href="https://elixir.bootlin.com/linux/v4.8/ident/target"><strong>target</strong></a>, <a href="https://elixir.bootlin.com/linux/v4.8/ident/ipi_types"><strong>ipi_types</strong></a>[ipinr]);<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/__smp_cross_call"><strong>__smp_cross_call</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/target"><strong>target</strong></a>, ipinr);<br>}</pre><p>怎麼知道註冊IPI_RESCHEDULE 的function是誰呢？還好同樣在smp.c 裡看到了這個 set_smp_cross_call() , 所以只要找找誰有透過set_smp_cross_call() 註冊 callback function 就行了</p><pre>void <a href="https://elixir.bootlin.com/linux/v4.8/ident/__init"><strong>__init</strong></a> <a href="https://elixir.bootlin.com/linux/v4.8/ident/set_smp_cross_call"><strong>set_smp_cross_call</strong></a>(void (*fn)(const struct <a href="https://elixir.bootlin.com/linux/v4.8/ident/cpumask"><strong>cpumask</strong></a> *, unsigned int))<br>{<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/__smp_cross_call"><strong>__smp_cross_call</strong></a> = fn;<br>}</pre><p>找到了！ gic driver 在init 時會call set_smp_cross_call(gic_raise_softirq), 所以smp_send_reschedule() 最終會走到gic_raise_softirq(), 而這邊會算出cluster id 後帶入gic_send_sgi(), 繼續往下看</p><pre>static void <a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_raise_softirq"><strong>gic_raise_softirq</strong></a>(const struct <a href="https://elixir.bootlin.com/linux/v4.8/ident/cpumask"><strong>cpumask</strong></a> *mask, unsigned int irq)<br>{<br>	int cpu;<br><br>	if (<a href="https://elixir.bootlin.com/linux/v4.8/ident/WARN_ON"><strong>WARN_ON</strong></a>(irq &gt;= 16))<br>		return;<br><br>	<em>/*</em><br><em>	 * Ensure that stores to Normal memory are visible to the</em><br><em>	 * other CPUs before issuing the IPI.</em><br><em>	 */</em><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_wmb"><strong>smp_wmb</strong></a>();<br><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/for_each_cpu"><strong>for_each_cpu</strong></a>(cpu, mask) {<br>		unsigned long cluster_id = <a href="https://elixir.bootlin.com/linux/v4.8/ident/cpu_logical_map"><strong>cpu_logical_map</strong></a>(cpu) &amp; ~0xffUL;<br>		u16 tlist;<br><br>		tlist = <a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_compute_target_list"><strong>gic_compute_target_list</strong></a>(&amp;cpu, mask, cluster_id);<br>		<a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_send_sgi"><strong>gic_send_sgi</strong></a>(cluster_id, tlist, irq);<br>	}<br><br>	<em>/* Force the above writes to ICC_SGI1R_EL1 to be executed */</em><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/isb"><strong>isb</strong></a>();<br>}<br>static void <a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_smp_init"><strong>gic_smp_init</strong></a>(void)<br>{<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/set_smp_cross_call"><strong>set_smp_cross_call</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_raise_softirq"><strong>gic_raise_softirq</strong></a>);<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/cpuhp_setup_state_nocalls"><strong>cpuhp_setup_state_nocalls</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/CPUHP_AP_IRQ_GICV3_STARTING"><strong>CPUHP_AP_IRQ_GICV3_STARTING</strong></a>,<br>				  &quot;AP_IRQ_GICV3_STARTING&quot;, <a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_starting_cpu"><strong>gic_starting_cpu</strong></a>,<br>				  NULL);<br>}</pre><p>一樣是在 kernel-4.9/drivers/irqchip/irq-gic-v3.c 可以看到 gic_send_sgi 的實作. 這邊為了debug 而呼叫了 smp_processor_id(), 不過實際上還是用其他的方式指定要叫哪個cpu 起來 (看起來是將 cluster id 組合成val 直接填入gic register), 也就是說喚醒cpu 的過程沒有問題, 那smp_processor_id() 到底為什麼會印log?</p><pre>static void <a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_send_sgi"><strong>gic_send_sgi</strong></a>(u64 cluster_id, u16 tlist, unsigned int irq)<br>{<br>	u64 val;<br><br>	val = (<a href="https://elixir.bootlin.com/linux/v4.8/ident/MPIDR_TO_SGI_AFFINITY"><strong>MPIDR_TO_SGI_AFFINITY</strong></a>(cluster_id, 3)	|<br>	       <a href="https://elixir.bootlin.com/linux/v4.8/ident/MPIDR_TO_SGI_AFFINITY"><strong>MPIDR_TO_SGI_AFFINITY</strong></a>(cluster_id, 2)	|<br>	       irq &lt;&lt; <a href="https://elixir.bootlin.com/linux/v4.8/ident/ICC_SGI1R_SGI_ID_SHIFT"><strong>ICC_SGI1R_SGI_ID_SHIFT</strong></a>		|<br>	       <a href="https://elixir.bootlin.com/linux/v4.8/ident/MPIDR_TO_SGI_AFFINITY"><strong>MPIDR_TO_SGI_AFFINITY</strong></a>(cluster_id, 1)	|<br>	       tlist &lt;&lt; <a href="https://elixir.bootlin.com/linux/v4.8/ident/ICC_SGI1R_TARGET_LIST_SHIFT"><strong>ICC_SGI1R_TARGET_LIST_SHIFT</strong></a>);<br><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/pr_debug"><strong>pr_debug</strong></a>(&quot;CPU%d: ICC_SGI1R_EL1 %llx\n&quot;, <a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_processor_id"><strong>smp_processor_id</strong></a>(), val);<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/gic_write_sgi1r"><strong>gic_write_sgi1r</strong></a>(val);<br>}</pre><p>smp_processor_id() 被defined 於 kernel-4.9/include/linux/smp.h. 在我們的例子裡, CONFIG_DEBUG_PREEMPT 有enable, 因此會呼叫 debug_smp_process_id()</p><pre><em>/*</em><br><em> * smp_processor_id(): get the current CPU ID.</em><br><em> *</em><br><em> * if DEBUG_PREEMPT is enabled then we check whether it is</em><br><em> * used in a preemption-safe way. (smp_processor_id() is safe</em><br><em> * if it&#39;s used in a preemption-off critical section, or in</em><br><em> * a thread that is bound to the current CPU.)</em><br><em> *</em><br><em> * NOTE: raw_smp_processor_id() is for internal use only</em><br><em> * (smp_processor_id() is the preferred variant), but in rare</em><br><em> * instances it might also be used to turn off false positives</em><br><em> * (i.e. smp_processor_id() use that the debugging code reports but</em><br><em> * which use for some reason is legal). Don&#39;t use this to hack around</em><br><em> * the warning message, as your code might not work under PREEMPT.</em><br><em> */</em></pre><pre>#ifdef CONFIG_DEBUG_PREEMPT<br>  extern unsigned int <a href="https://elixir.bootlin.com/linux/v4.8/ident/debug_smp_processor_id"><strong>debug_smp_processor_id</strong></a>(void);<br># define <a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_processor_id"><strong>smp_processor_id</strong></a>() <a href="https://elixir.bootlin.com/linux/v4.8/ident/debug_smp_processor_id"><strong>debug_smp_processor_id</strong></a>()<br>#else<br># define <a href="https://elixir.bootlin.com/linux/v4.8/ident/smp_processor_id"><strong>smp_processor_id</strong></a>() <a href="https://elixir.bootlin.com/linux/v4.8/ident/raw_smp_processor_id"><strong>raw_smp_processor_id</strong></a>()<br>#endif</pre><p>debug_smp_processor_id() 會呼叫check_preemption_disabled(), code 實作於kernel-4.9/lib/smp_processor_id.c. 這個function就是在取得 cpu id 之前做一些檢查</p><ol><li>此時preemption是否被disabled?</li><li>此時irq 是否被disabled?</li><li>此thread是否有binding到特定cpu</li></ol><p>如果都不是就會印出下面的warning log 和吐出一堆backtrace…</p><p>BUG: using smp_processor_id() in preemptible</p><p>至於為什麼要做檢查呢？ 因為this_cpu 是由當下thread的info中取得的. 若此時thread 被preempt出去, 又被其他cpu執行, 則有可能會取得錯誤的this_cpu 資訊</p><pre><a href="https://elixir.bootlin.com/linux/v4.8/ident/notrace"><strong>notrace</strong></a> static unsigned int <a href="https://elixir.bootlin.com/linux/v4.8/ident/check_preemption_disabled"><strong>check_preemption_disabled</strong></a>(const char *what1,<br>							const char *what2)<br>{<br>	int <a href="https://elixir.bootlin.com/linux/v4.8/ident/this_cpu"><strong>this_cpu</strong></a> = <a href="https://elixir.bootlin.com/linux/v4.8/ident/raw_smp_processor_id"><strong>raw_smp_processor_id</strong></a>();<br><br>	if (<a href="https://elixir.bootlin.com/linux/v4.8/ident/likely"><strong>likely</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/preempt_count"><strong>preempt_count</strong></a>()))<br>		goto out;<br><br>	if (<a href="https://elixir.bootlin.com/linux/v4.8/ident/irqs_disabled"><strong>irqs_disabled</strong></a>())<br>		goto out;<br><br>	<em>/*</em><br><em>	 * Kernel threads bound to a single CPU can safely use</em><br><em>	 * smp_processor_id():</em><br><em>	 */</em><br>	if (<a href="https://elixir.bootlin.com/linux/v4.8/ident/cpumask_equal"><strong>cpumask_equal</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/tsk_cpus_allowed"><strong>tsk_cpus_allowed</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/current"><strong>current</strong></a>), <a href="https://elixir.bootlin.com/linux/v4.8/ident/cpumask_of"><strong>cpumask_of</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/this_cpu"><strong>this_cpu</strong></a>)))<br>		goto out;<br><br>	<em>/*</em><br><em>	 * It is valid to assume CPU-locality during early bootup:</em><br><em>	 */</em><br>	if (<a href="https://elixir.bootlin.com/linux/v4.8/ident/system_state"><strong>system_state</strong></a> != <a href="https://elixir.bootlin.com/linux/v4.8/ident/SYSTEM_RUNNING"><strong>SYSTEM_RUNNING</strong></a>)<br>		goto out;<br><br>	<em>/*</em><br><em>	 * Avoid recursion:</em><br><em>	 */</em><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/preempt_disable_notrace"><strong>preempt_disable_notrace</strong></a>();<br><br>	if (!<a href="https://elixir.bootlin.com/linux/v4.8/ident/printk_ratelimit"><strong>printk_ratelimit</strong></a>())<br>		goto out_enable;<br><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/printk"><strong>printk</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/KERN_ERR"><strong>KERN_ERR</strong></a> &quot;BUG: using %s%s() in preemptible [%08x] code: %s/%d\n&quot;,<br>		what1, what2, <a href="https://elixir.bootlin.com/linux/v4.8/ident/preempt_count"><strong>preempt_count</strong></a>() - 1, <a href="https://elixir.bootlin.com/linux/v4.8/ident/current"><strong>current</strong></a>-&gt;<a href="https://elixir.bootlin.com/linux/v4.8/ident/comm"><strong>comm</strong></a>, <a href="https://elixir.bootlin.com/linux/v4.8/ident/current"><strong>current</strong></a>-&gt;<a href="https://elixir.bootlin.com/linux/v4.8/ident/pid"><strong>pid</strong></a>);<br><br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/print_symbol"><strong>print_symbol</strong></a>(&quot;caller is %s\n&quot;, (long)__builtin_return_address(0));<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/dump_stack"><strong>dump_stack</strong></a>();<br><br>out_enable:<br>	<a href="https://elixir.bootlin.com/linux/v4.8/ident/preempt_enable_no_resched_notrace"><strong>preempt_enable_no_resched_notrace</strong></a>();<br>out:<br>	return <a href="https://elixir.bootlin.com/linux/v4.8/ident/this_cpu"><strong>this_cpu</strong></a>;<br>}<br><br><a href="https://elixir.bootlin.com/linux/v4.8/ident/notrace"><strong>notrace</strong></a> unsigned int <a href="https://elixir.bootlin.com/linux/v4.8/ident/debug_smp_processor_id"><strong>debug_smp_processor_id</strong></a>(void)<br>{<br>	return <a href="https://elixir.bootlin.com/linux/v4.8/ident/check_preemption_disabled"><strong>check_preemption_disabled</strong></a>(&quot;smp_processor_id&quot;, &quot;&quot;);<br>}<br><a href="https://elixir.bootlin.com/linux/v4.8/ident/EXPORT_SYMBOL"><strong>EXPORT_SYMBOL</strong></a>(<a href="https://elixir.bootlin.com/linux/v4.8/ident/debug_smp_processor_id"><strong>debug_smp_processor_id</strong></a>);</pre><p>現在已經知道為什麼會有BUG 的log出現, solution 就是call smp_processor_id() 之前一定要確保有符合那三項條件, 最簡單的方式就是call preempt_disable/preempt_enable</p><p>有空再來深究 this_cpu 的取法以及preempt 的用法吧</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=489f37ddf421" width="1" height="1" alt=""><hr><p><a href="https://medium.com/what-notes/dump-error-log-while-calling-smp-proccessor-id-489f37ddf421">Dump error log while calling smp_proccessor_id()</a> was originally published in <a href="https://medium.com/what-notes">What? Notes!</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>