<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Junbum Lee (Beomi) on Medium]]></title>
        <description><![CDATA[Stories by Junbum Lee (Beomi) on Medium]]></description>
        <link>https://medium.com/@JunbumLee?source=rss-395330a2e7cc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*YzsdioQzMLLX2KsBnnKUlQ.png</url>
            <title>Stories by Junbum Lee (Beomi) on Medium</title>
            <link>https://medium.com/@JunbumLee?source=rss-395330a2e7cc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:14:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@JunbumLee/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[dLLM into TPU: An End-to-End Diffusion LM Stack in Pure JAX]]></title>
            <link>https://medium.com/@JunbumLee/dllm-into-tpu-an-end-to-end-diffusion-lm-stack-in-pure-jax-5fc33c840ebb?source=rss-395330a2e7cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fc33c840ebb</guid>
            <category><![CDATA[jax]]></category>
            <category><![CDATA[tpusprint]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[tpu]]></category>
            <category><![CDATA[diffusion-llm]]></category>
            <dc:creator><![CDATA[Junbum Lee (Beomi)]]></dc:creator>
            <pubDate>Fri, 17 Apr 2026 15:32:50 GMT</pubDate>
            <atom:updated>2026-04-17T15:32:50.212Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From an autoregressive HuggingFace checkpoint to a KV-cached Soft Parallel Decoder on TPU v4/v5/v6— without PyTorch. #TPUSprint #TRC</em></p><blockquote><strong>Github Repo: </strong><a href="https://github.com/Beomi/dllm-jax"><strong>https://github.com/Beomi/dllm-jax</strong></a></blockquote><h3>Why this matters</h3><p>Diffusion Language Models (dLLMs) are the most interesting alternative paradigm to autoregressive LLMs right now. Papers like MDLM, BD3LM, Dream, and DMax have shown that masked-diffusion objectives can match or beat left-to-right decoding on reasoning benchmarks — while decoding <em>blocks</em> in parallel instead of tokens sequentially. The catch: almost every reference implementation ships as a PyTorch + CUDA training recipe with a trust-remote-code HuggingFace wrapper around a specific base model. Running them on TPU, against your own checkpoint, without dragging in CUDA-only kernels, is a hand-rolled project.</p><p><a href="https://github.com/beomi/dllm-jax"><strong>dllm-jax</strong></a> closes that loop end-to-end. You can take a <em>standard autoregressive HuggingFace checkpoint</em> — Currently Qwen3 supported — and push a single codebase from pretrained weights → OPUT fine-tuning on TPU v4/v5/v6 → KV-cached Soft Parallel Decoding inference, all in JAX/Flax. Zero PyTorch, zero CUDA, zero trust-remote-code model subclass.</p><p>What’s unusual here:</p><ol><li>It’s <strong>diffusion</strong>, not autoregressive. Block-causal attention with a [noised | clean] two-stream layout at training time; parallel decoding with confidence-weighted soft embeddings at inference.</li><li>It implements <strong>very fresh dLLM research</strong> — DMax’s On-Policy Uniform Training (OPUT) and Soft Parallel Decoding (SPD), with the associated paper literally only a week old at the time of porting — mapping the PyTorch reference (<a href="https://github.com/czg1225/DMax">czg1225/DMax</a>) line-by-line onto JAX primitives.</li><li>It’s <strong>end-to-end and reproducible on TPU</strong>, with a real KV-cache path, not a benchmark demo.</li></ol><h3>The pipeline</h3><figure><img alt="HF autoregressive ckpt (Qwen3–8B, safetensors) │ │ torch-free loader: safetensors + numpy + huggingface_hub ▼ Flax NNX model graph │ │ 2D FSDP + TP sharding, Pallas flash attention, │ CPU-first init + remat for 8B+ ▼ DMax OPUT training loop │ · two-stream [noised | clean] │ · block-diffusion attention mask │ · per-step on-policy rollout (stop_gradient forward → argmax → re-noise) │ · gradient accumulation, cosine LR │ │ Orbax DCP sharded checkpoints → GCS (us-central2, us-east1," src="https://cdn-images-1.medium.com/max/1024/1*5JeMPZgr8dhaAbyO9jWN2A.png" /><figcaption>The pipeline of dllm-jax</figcaption></figure><p>Everything in that chain ships in one repo. The same Python call signature (build_model_from_pretrained(&quot;Qwen/Qwen3-8B&quot;, task=&quot;llada&quot;)) starts you at the top; a single bash command (INFER_IMPL=kv_fast … python3 scripts/tpu_dmax_infer_checkpoint.py) finishes you at the bottom.</p><h3>What’s “DMax” and why port it?</h3><p>DMax (Aggressive Parallel Decoding for dLLMs, <a href="https://arxiv.org/abs/2604.08302">https://arxiv.org/abs/2604.08302</a> / the paper is a week old at time of writing) proposes two pieces:</p><p><strong>OPUT (On-Policy Uniform Training).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oKx4pIkF63CyD49u2GZ3hA.png" /><figcaption>Overview of the DMax’s On-Policy Uniform Training. (img from paper)</figcaption></figure><p>Classic masked-diffusion trainers sample a noise level, mask tokens, and predict x₀. DMax says: on every step, also run the model <em>under </em><em>stop_gradient</em>, argmax its predictions, and substitute those semi-predictions back into the masked positions before the loss forward. The model sees its own outputs on roughly half the batches. This closes the training-inference gap that vanilla MDLM suffers from at block-wise decoding time.</p><p><strong>SPD (Soft Parallel Decoding).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqsAYmIMlPvwrH_ympXJ1A.png" /><figcaption>Overview of the Soft Parallel Decoding procedure in DMax. (img from paper)</figcaption></figure><p>At inference, generation happens block-by-block. Inside a block, at each denoising step, the model looks at positions that have already been decoded and feeds them back — not as hard token embeddings, but as <em>confidence-weighted mixes</em> of the top-k predicted token embeddings with the mask embedding:</p><pre>soft_embed[P] = Σₖ topk_prob[P, k] · embed(topk_idx[P, k])<br>                + (1 − Σₖ topk_prob[P, k]) · mask_embed</pre><p>then renormalized to the probability-weighted target norm. Positions that passed a left-to-right confidence threshold get committed. The block loops until “nothing changed this step” or “every active position exceeds 0.9 confidence.”</p><p>This recipe outperforms naive masked-diffusion decoding substantially on reasoning benchmarks. On our own Qwen3–8B checkpoint trained with DMax OPUT on TinyStories, the difference between top_k=1 (reference default) and top_k=3 is the difference between &quot;her her her her…&quot; collapse and a complete multi-paragraph story that ends with &quot;The end.&quot; — same weights, same prompt, same threshold.</p><p>Porting this to JAX is not a mechanical exercise. The reference uses torch.compile, Python while-loops with break, PyTorch HF KV-cache plumbing, jax.lax-free control flow — every one of those patterns needs a different strategy under XLA.</p><h3>Three SPD implementations — and why</h3><p>The repo ships <em>three</em> Soft Parallel Decoders. They produce <strong>byte-exact identical output</strong> at matching settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*68hDI2hFGdSu43PaejtPBg.png" /></figure><p>Why three? Because each one is verifying the others. The Python-loop path is the human-readable reference. The fast path is what you actually run — fully traced, step-level while_loop with (step_index &lt; block_length) &amp; ~jnp.all(step_done) as the condition. The kv_fast path is where the real engineering is: we had to add KV cache <em>to the model</em> — not just the SPD loop — because until this session GenericDecoderLM only had a full-sequence forward.</p><h3>The KV-cache port</h3><p>Reference’s ThresholdParallelDecoder reuses the HuggingFace past_key_values protocol: at each step, forward only the current block&#39;s tokens; the HF model internally slots new K/V into the cache at replace_position. In JAX we don&#39;t have HF&#39;s runtime — we had to build it.</p><p>Three coordinated changes in dllm_jax/models.py:</p><pre>class SelfAttention(nnx.Module):<br>    def call_cached(self, hidden_states, past_k, past_v, cache_position, *,<br>                    attention_mask, position_ids):<br>        q, k_new, v_new = self._project_qkv(hidden_states, position_ids)<br>        past_k_updated = jax.lax.dynamic_update_slice(<br>            past_k, k_new.astype(past_k.dtype), (0, cache_position, 0, 0))<br>        past_v_updated = jax.lax.dynamic_update_slice(<br>            past_v, v_new.astype(past_v.dtype), (0, cache_position, 0, 0))<br>        q = q.astype(past_k.dtype)  # match QK-norm&#39;s float32 output<br>        output = self._attention(q, past_k_updated, past_v_updated, attention_mask)<br>        return output, past_k_updated, past_v_updated</pre><p>TransformerBlock.call_cached and GenericDecoderLM.call_cached plumb these through. The cache is a Python list of (past_k, past_v) tuples, one per layer, each shape [B, max_seq, num_kv_heads, head_dim] — JAX traces through it as a PyTree fine. Default behavior (no KV cache) is <em>byte-exact</em> unchanged.</p><h3>Reference’s “cross-block update” in JAX</h3><p>One subtle piece: during a block’s denoising loop, the K/V written to cache for the active block is <em>soft</em> (it came from the soft-mix inputs). When we move to the next block, we need <em>hard</em> K/V for the just-committed block. Reference combines this with the next block’s first step as a 2-block forward (is_cross_block=True). Under static XLA shapes that&#39;s awkward.</p><p>Our approach: <strong>post-block hard-write</strong>. After each block’s while-loop terminates, run one more forward with embed(committed_tokens) as input, write K/V to the same cache position. One extra forward per block; same semantic result:</p><pre>final_block = jax.lax.dynamic_slice(x_new, (0, block_start), (batch_size, block_length))<br>hard_embeds = _embed_tokens(current_model, final_block)<br>hard_out = current_model.call_cached(<br>    inputs_embeds=hard_embeds,<br>    past_key_values=past_kv_new,<br>    cache_position=block_start,<br>    attention_mask=attn_mask_block,<br>    position_ids=block_pos_ids,<br>)<br>past_kv_hard = hard_out[&quot;past_key_values&quot;]</pre><p>The gain is real: at 1024-token generation we measured kv_fast at <strong>76.6 s</strong> vs fast at <strong>124.7 s</strong> on TPU v4-32, with byte-exact identical generated text.</p><h3>End-to-end, measured</h3><p>Verified on tpu-v4-32 against an 8B Qwen3 we fine-tuned with DMax OPUT on TinyStories. These are real numbers, not whiteboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2Wkm-24faw1s6yjKPk5Yg.png" /></figure><p>The same kv_fast path on the latest checkpoint (step 19100) produces:</p><blockquote>Once upon a time, there was a little girl named Lily. She loved to play with her toys and run around outside. One day, she went to the park with her mom and dad. They played on the swings and went down the slide. After playing for a while, Lily’s mom said it was time to go home… She closed her eyes and fell asleep. The end.</blockquote><p>nfe=94, generated_tokens=137. The model emitted EOS early, the outer while_loop short-circuited, and _trim_generated_tokens cut at EOS — exactly mirroring reference&#39;s early_stop post-EOS fill.</p><h3>So, How the training was?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U4vHB-zphJ19QyNq0VAI7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TOVa5OQ0HXKbzBUWznvRBg.png" /></figure><p>Note: First run (0 to 1.2k steps) was trained on TPUv6e-64, and later it was resumed at TPUv5e-64. That makes significant difference between tokens_per_second metrics.</p><p>The first 1,200 steps ran on TPU v6e-64; the remainder resumed on TPU v5e-64 — the hardware switch accounts for the visible discontinuity in tokens/second metrics. The training corpus was <a href="https://huggingface.co/datasets/roneneldan/TinyStories">TinyStories</a>, covering roughly 1.4 epochs. Loss and gradient norm start high and converge steadily — standard behavior for a cold start on a pretrained base.</p><h3>The full stack you’d actually need</h3><p>Running dLLM research on TPU tends to fall apart at the boundaries:</p><ul><li><strong>“How do I even load the base model?”</strong> — safetensors + numpy + huggingface_hub, no torch.load; Qwen3 Dense shapes map straight into Flax NNX. build_model_from_pretrained(&quot;Qwen/Qwen3-8B&quot;, task=&quot;llada&quot;) returns a ready-to-train NNX model.</li><li><strong>“How do I shard 8B on TPU?”</strong> — 2D FSDP+TP mesh ((ndev/tp, tp) axes fsdp,tp), CPU-first init via jax.default_device(jax.devices(&quot;cpu&quot;)[0]), jax.remat per transformer layer, Pallas flash attention via shard_map.</li><li><strong>“How do I survive multi-host checkpointing?”</strong> — Orbax StandardCheckpointer / PyTreeCheckpointHandler with MultiprocessingOptions(primary_host=0), barrier_sync_fn wired to multihost_utils.sync_global_devices, per-step commit_success.txt markers, and a latest_committed_gcs_step scanner for robust resume. Training and inference meshes can differ — restore reshards automatically.</li><li><strong>“How do I run inference from a sharded GCS checkpoint?”</strong> — scripts/tpu_dmax_infer_checkpoint.py: multi-host SSH, Orbax restore, nnx.split / nnx.merge for reshard, then dmax_generate_spd_{fast,kv_fast,legacy} end-to-end. INFER_IMPL=kv_fast RESUME_DIR=gs://... TOP_K=3 TP=8 python3 ….</li></ul><p>The result: a commit-level-reproducible path from Qwen/Qwen3-8B → DMax-trained diffusion LM → KV-cached parallel decoding on TPU.</p><h3>Reference-alignment receipts</h3><p>We didn’t just port vibes — we verified the port line-by-line against the reference decoder:</p><ul><li>get_transfer_index_uniform&#39;s left-to-right confident-mask selection with first-mask fallback: _select_leftmost_confident_masks.</li><li>decode_uniform&#39;s update rule update_mask = high_conf_index | (active_index &amp; ~mask_index): same where-stack in the step body.</li><li>Break conditions max_probs[active] &gt;= 0.9 or not changed_mask.any(): same.</li><li>x0_p threshold semantics at temperature &gt; 0 (sampled-token probability, not argmax): _sample_x0 returns chosen_probs for threshold and max_probs for confidence_stop, exactly matching.</li><li>Soft-mix renormalization target_norm / (current_norm + 1e-6) (parallel_strategy.py line 656): same additive epsilon, not a clamp.</li><li>add_gumbel_noise&#39;s exp(logits) / (-log(U))^T formulation reduces to argmax(logits/T + gumbel) — our implementation is equivalent up to argmax.</li><li>Reference’s early_stop post-EOS fill orig_x[eos_idx, block_loc.end:] = eos_id: matched via jnp.where(fill_cond, eos_id, cur_x).</li></ul><p>Internal consistency is equally strong: fast ≡ kv_fast byte-exact at 128 tokens (identical generated text); at 1024 tokens the only diff in the output logs is the nfe / generate_seconds metric lines.</p><h3>Takeaways</h3><ul><li><strong>End-to-end matters.</strong> Most open diffusion-LM releases ship a training recipe and a demo, leaving “run it against your own model on your own hardware” as an exercise. dllm-jax treats that as the headline feature.</li><li><strong>Latest research, on day one.</strong> DMax is a week old. OPUT + SPD + KV-cache are all in the repo. You can read the paper on Monday and train it on Tuesday.</li><li><strong>Diffusion LMs belong on TPU.</strong> The block-parallel decode pattern maps unusually well to TPU: static shapes, while_loop-friendly step counts, KV cache as a fixed [B, total_length, num_kv_heads, head_dim] buffer. The 1.63× KV-cache speedup comes free from this geometry.</li><li><strong>Algorithmic equivalence &gt; rewriting from scratch.</strong> By mapping the reference’s Python line numbers to JAX primitives one-to-one (with jax.lax.while_loop for Python break, dynamic_update_slice for replace_position, post-block hard-write for cross-block update), you get a port that is both defensibly correct and XLA-friendly. That&#39;s the part that makes it reproducible research rather than a demo.</li></ul><p><em>The code, design docs, and TPU runbook are in the </em><a href="https://github.com/Beomi/dllm-jax"><em>repository</em></a><em>.<br>Thanks to TRC Team for granting TPU for this research.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fc33c840ebb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Django 스터디] 입문 2주차 | Django MTV?]]></title>
            <link>https://medium.com/djangogirlsseoul-codecamp/django-%EC%8A%A4%ED%84%B0%EB%94%94-%EC%9E%85%EB%AC%B8-2%EC%A3%BC%EC%B0%A8-django-mtv-d0c033ef9dca?source=rss-395330a2e7cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0c033ef9dca</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[django-girls]]></category>
            <dc:creator><![CDATA[Junbum Lee (Beomi)]]></dc:creator>
            <pubDate>Fri, 17 Feb 2017 10:29:27 GMT</pubDate>
            <atom:updated>2017-02-22T05:23:22.985Z</atom:updated>
            <content:encoded><![CDATA[<p>Django Girls Seoul에서 Facilitator로 활동하며 Django 입문 스터디 기획을 맡은 이준범(@<a href="https://medium.com/u/395330a2e7cc">Beomi</a>)입니다.</p><p>처음 스터디 모집때 스터디의 주제는 ‘간단한 API 만들어보기’ 였지만, 첫 모임에서 각자 원하는 팀 별 주제를 정해 ‘블로그’, ‘인스타그램’, ‘유저 관리 서비스’등의 주제를 통해 프로젝트를 진행하며 스터디에서 배운 것을 활용하는 방식으로 진행하기로 하였습니다.</p><h3><strong>첫 만남</strong></h3><p>2월 5일 Django 입문 스터디로 스터디원 분들을 만나보았습니다. 첫 날임에도 Django에 대한 열정이 가득해 앞으로 스터디에서 다룰 주제, 내용에 대해 이야기 해보는 시간을 가졌습니다.</p><h4>기획하기</h4><p>어떤 프로젝트를 진행하자! 라고 할 때, 가장 먼저 해야하는 것은 바로 ‘어떤 것을 만들지’ 정하는 것입니다. 첫 만남에서는 우리가 만들 블로그와 인스타그램 등에서 어떤 기능을 제공해야하는지에 대해 논의하는 시간을 가졌습니다.</p><p>블로그 팀은 Django Girls Tutorial에서 다루는 블로그와 유사하게 게시글 위주로 블로그를 만들어 보기로 하였고, 인스타그램팀은 인스타그램을 벤치마킹해 좋아요/댓글 등의 기능까지 구현해보는 것을 고려하고 아래의 강좌를 따라가며 Django를 학습해보기로 하였습니다.</p><p><a href="http://blog.hannal.com/category/start-with-django-webframework/">Category: start-with-django-webframework · Kay on the rails</a></p><h3>두번째 모임</h3><p>이번 스터디는 책이나 강좌를 따라가는 방식이 아니라 프로젝트 방식이기 때문에 각 팀별로 주제를 정하고 각 개인의 Github Repository에서 각자의 Django Project를 만들어서 배포까지 하는 것을 목표로 삼고 있습니다.</p><p><a href="https://github.com/djangogirlsseoul2017">DjangoGirls Seoul 2017 · GitHub</a></p><h4>Django MTV의 기초</h4><p>두번째 모임에서는 Django MTV와 RESTFul에 대한 가벼운 설명으로 시작했습니다.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Fembed_code%2Fkey%2FeCRYXTNp3MniB4&amp;url=https%3A%2F%2Fwww.slideshare.net%2FJunbumLee%2Fdjangogirls-seoul-django-study-django-modelviewtemplate&amp;image=https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2F01-djangomtv-r-170217100314-thumbnail-4.jpg%3Fcb%3D1487325864&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=slideshare" width="600" height="500" frameborder="0" scrolling="no"><a href="https://medium.com/media/55e63c36c65affe2a22c7996e76c2fa1/href">https://medium.com/media/55e63c36c65affe2a22c7996e76c2fa1/href</a></iframe><h4>Models.py 설계 &amp; 코딩</h4><p>Django에서 Models는 가장 핵심인 부분이며, 기획의 기능을 구현하는 대다수가 이 부분에서 이뤄집니다.<br>그림을 그려 관계를 설정하고, 어떤 요소가 우리가 기획하는 프로젝트에 필요한지를 고민하는 시간을 가져보았습니다.</p><blockquote>“중요한 것은 질문을 멈추지 않는 것입니다.”</blockquote><h3>질문하기</h3><p>질문을 한다는 것은 내용에 대한 이해 뿐 아니라 용기도 필요합니다.<br>모집을 할 때 수준 편차를 줄이려 노력하지만 실제로 둘 이상의 사람만 있어도 수준의 편차가 나기 때문에 어떤 질문을 할 때는 스스로 ‘이 질문은 너무 쉬운 질문이 아닌가?’ 하고 생각할 수 있습니다. 하지만 질문을 하고 답하는 방법은 학습에 있어 최고의 이해를 이끌어낼 수 있기 때문에, 자유롭게 질문을 하고 상대의 질문에 답을 생각해 보는 것이 독학이 아닌 스터디를 통해 얻을 수 있는 큰 장점이라 생각합니다.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0c033ef9dca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/djangogirlsseoul-codecamp/django-%EC%8A%A4%ED%84%B0%EB%94%94-%EC%9E%85%EB%AC%B8-2%EC%A3%BC%EC%B0%A8-django-mtv-d0c033ef9dca">[Django 스터디] 입문 2주차 | Django MTV?</a> was originally published in <a href="https://medium.com/djangogirlsseoul-codecamp">Django Girls Seoul | 장고걸스서울</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>