TOPPAN様との協業をご紹介いただきました！ 世界最強のHuman-in-the-loopのデータ構造化基盤を作っていきます💪

「ぱっと見AIだとわからない画像の見破り方」 AI画像生成は３D幾何学的な理解を苦手とするのでこんな感じに障害物の裏で続いてるはずの線を書くと矛盾が生じているのが一目で分かる。実はRANSACとハフ変換といった学部レベルのコンピュータビジョン技術である程度自動的に見破れる！

Using a vision transformer and a speaker to vibrate a bottle to see inside of it

Anthropic 東京拠点の新設を記念して、イベントをやるそうなので興味ある方はご連絡ください〜

こうやって皆で寄ってたかって叩くのもどうかと思うなあ。結果が間違ってるなら“これだから日本はダメなんだ”とか“バカ研究”とか無駄に中傷を混ぜて拡散させるよりも直に著者に間違いを教えてあげればいいと思う。別分野の研究者が積極的に深層学習を取り入れようと頑張ってるのは喜ばしいことなのに。

CESで今日発表されたNVIDIAの「世界基盤AI」がもうすでにオープンソース（Apache-2）で出ている！

CAD-Llama 🦙⚙️ Converts the DeepCAD dataset into an LLM-friendly text format and instruction fine-tunes a Llama-8B model for CAD generation and editing. Accuracy increases 4x relative to vanilla GPT4, 2x relative to instruction fine-tuning on a more 'raw' CAD representation

📢 Our new @NVIDIAAI paper is up on arXiv! I'm happy to share Neural Geometric Level of Detail, which enables the first ⚡real-time rendering of high quality neural #3d SDFs with a sparse octree feature volume and a tiny MLP! nv-tlabs.github.io/nglod/

創業した会社がY Combinatorに採択されました！！NVIDIAを退職してから色々と波瀾万丈な人生ですがこれからもよろしくお願いします。 @outerbasisは「AIモデル管理」の会社です。社内用LLMの読み込みを高速化したり、セキュリティ対策を万全にします。社会インフラ構築を目指して頑張ります！！

Dataset of 100k STL files and G-code for 3D printing

I had a day to procrastinate over the holidays, so I made an open source research-oriented portfolio website template in Next.js and React- where you can update everything just by editing TypeScript objects and looks good on mobile

Great slides on verifiers for reasoning LLMs

Adding horizontal lines to images improves VLM (vision language model) performance of tasks like counting, visual search, spatial understating, scene understanding, and more

もうそろそろGaussian Splat+三次元生成が出てくるだろうと先週、話してたらまさに今出てきた: dreamgaussian.github.io/ (画像からメッシュ生成が2分でできる！)

カーネギーメロン大学の「LLMシステム基盤」の講義。 GPUとかトランスフォーマの基礎から始まって、分散学習、ロングコンテキスト、推論的デコーディングなどもカバー。スライドも全部公開しているのが嬉しい！ （個人的には推論系論文もっと取り上げてほしい） llmsystem.github.io/llmsyste…

正式にUniversity of Toronto CS PhD進学が決まりました！院では引き続き謎の半導体企業と共に機械学習 x グラフィックスの研究をする予定です。 ついでにカナダのCS大学院についてまとめてみたので良ければご覧下さい。秋からトロントにいるのでよろしくお願いします🙇‍♂️ medium.com/@yongyuanxi/%E3%8…

Yeah document parsing is cool, but what about CAD drawing parsing... (but with documents too 🥺)

APT 🌹: an adaptive quadtree-like patching scheme for more efficient vision transformers. Speeds up inference and training by ~40% (with higher relative accuracy compared to just resizing)

Drawing2CAD🖌️ Transformer-based AI model that converts *vector* drawings (SVGs) to a DeepCAD sequence. The use of vector input over raster (image) inputs results in ~7% higher accuracy

Creating CAD directly from images (dimension sketches), point cloud, and text. This is a model from Spectral Labs - I've had the chance to experiment with their model and it's great stuff 😎

大学生エミュするために久しぶりにカフェで勉強（仕事）しに行ったらChatGPTを開きながら作業する学生を大量に観測した

Jensen revealed the @NVIDIAAI text-to-3D (mesh models!) strategy at the GTC Keynote today. I'm super proud to be a part of this epic effort alongside my amazing coworkers 😊 Find out more here: nvidia.com/en-us/gpu-cloud/p…

"3D Object Reconstruction and Representation using Neural Networks" published in GRAPHITE 2004 by Peng et al. proposed neural implicit surfaces ages before anyone else did. Visionary work (even had a cat!) which doesn't seem to get the credit it should!

📢 Our new SIGGRAPH 2022 paper is on arXiv!! I'm happy to share Variable Bitrate Neural Fields, which lets you download ultra compact & high fidelity & fast NeRFs and neural fields at ⚡ blazing speeds with progressive LOD! 🌐: nv-tlabs.github.io/vqad/ 🎥: piped.video/watch?v=Lh0CoTRN…

【📢NEWS 】三次元スキャンなどに有用な高速かつ高性能なニューラル場 #neuralfields を構築することの出来るPyTorch研究ライブラリのNVIDIA Kaolin Wispをリリースしました！！Instant -NGPやNGLODなどをPyTorchでインタラクティブに学習・ぐりぐり動かすことが出来ます。 github.com/NVIDIAGameWorks/k…

LLMをツイッターのバズツイートを追加学習させると頭が悪くなり（リーズニング力が落ちる）と読解力（ロングコンテキスト理解力）が落ちることを示す論文

Detecting 3D lines from gaussian splats

テキストから３Dモデルのテキスチャを生成する論文が出てきた。いつかは出てくると思ってたけど相変わらず研究のペースが早い！ texturepaper.github.io/TEXTu…

Robots that navigate grocery stores with semantic 3D maps created by language-augmented gaussian splats

トロントで人気の大麻屋の名前は、なぜか“トーキョースモーク”

日本語版はまだないけど最近はHands-on Large Language Modelsを読むのが手っ取り早い気がする oreilly.com/library/view/han…

Predicting manufacturing costs directly from 2D CAD drawings using XGBoost on geometric features

📢 Our new library is on GitHub!! I'm happy to share NVIDIA Kaolin Wisp, a PyTorch library which provides building blocks and interactive apps to build your own complex neural fields. Presented with a nerf snippet of wisp development. 💻: github.com/NVIDIAGameWorks/k…

📢Our new @NVIDIAAI paper is out! I'm happy to share Compact NGP with Learned Hash Probing, which offers NeRFs that are 3.5x smaller than Instant NGP at a ~1.26x training cost and equivalent (or faster) inference speeds! 🌐: nv-tlabs.github.io/compact-n… 🎥: piped.video/watch?v=3TEry8zL…

Preferred Networks受かった！てなわけで今年は初めて日本に住む事になりそう。

📢Here's something cool we've been working on: efficient sparse octree primitives for 3D deep learning with PyTorch. This enables training NGLOD 3x faster with 30x less memory, which also means it can fit 30x more complex models! Code available here: github.com/nv-tlabs/nglod

I left my job at NVIDIA last year... and now I co-founded @outerbasis and started a position as ceo sararīman!! (also we're part of the current Y Combinator S24 batch 😎) With new tech come new formats, and with new formats come new distribution tech. We've seen this with cameras ➡️ JPEGs ➡️ social media and cgroups ➡️ containers ➡️ orchestration. Just like how cameras capture a slice of the photon stream around us, generative AI is a 'data camera' that allows us to snapshot the society that we live in. The new format from this capture device are model weights, which require tailored distribution solutions for them. @outerbasis is a registry for model weights that lets you store model weights and orchestrate the distribution of them onto GPU servers. The system uses tech optimized for streaming weights into the servers as fast as possible- so researchers can deploy updates without downtime and ops admins can orchestrate hot swapping between models. I've been super excited about streaming, compression, and neural networks for the last N years... but now I'm excited to build a system to support of this evermore exciting ecosystem of foundation model weights!! Please reach out if this sounds interesting / useful to you!! (and you can find more details on our YC launch: ycombinator.com/launches/La9…) (The video is a small demo of our model hotswapping)

Something cool that doesn't get talked enough is that we can do open vocabulary segmentation of 3D space

CVPR deadline life pro tip: you can directly generate LaTeX tables from Pandas pandas.pydata.org/pandas-doc…

Seek-CAD 🐳 DeepSeek R1-32B model for CAD generation _without_ fine-tuning through in-context learning and self-refinement via VLM (Gemini 2.0) feedback on the R1 CoT. Also a dataset with more ops than DeepCAD- chamfer, revolve, fillets, etc (but unfortunately closed source?)

Giving vision-language models the ability to reference the input image with segmentation masks

You Can't Manufacture a NeRF - an ICML position paper from CMU that discusses what it means to generate a 3D object that can be manufactured. Nothing too controversial and feels a bit more like a survey- but good reference to have for explaining this concept 🙂

Diffusion model to generate CAD sketches

5 years ago, we collected a small dataset of 3D SDFs. SDFs encode 3D surfaces via code that generate scalar fields- back then we ignored the "code" part and converted them into vector blobs. Today it feels painfully obvious that the real value was actually in the code.

🚨The source code for our #CVPR2021 Oral paper, Neural Geometric Level of Detail is now available on GitHub under the MIT License! GitHub: github.com/nv-tlabs/nglod Project Web: nv-tlabs.github.io/nglod/

Finding objects in 3D space using 3D segmentation masks and VLMs operating on multi-view image sequences.

End-to-end design and optimization of mechanical components with AI code generation in Rhino Grasshopper (with topology optimization, constraint setting)

🦖🦖🦖 Rex-Omni: an open-weight foundation model for text-to-bounding-box that can do object detection from prompt, or from examples.

We have openings for a research engineering internship at NVIDIA to work on open source software (eg github.com/NVIDIAGameWorks/k…) with us. If you want to contrib to cutting edge research for 3D generative modeling, editing and nerfs, send me a DM/email! Undergrads welcome :)

🗽MiCADangelo🗽: AI-assisted reverse engineering of physical objects (3D scan -> CAD). Employs a pretty unique approach via 1. CNN which selects sketch planes from proposals and 2. CNN for sketch parameterization (pretrained on SketchGraphs) and 3. differentiable extrusions

There's actually a lot of companies in East Asia that offer LiDAR hardware solutions alongside GS with BIM integration- CAD overlayed with well-calibrated / unwarped Gaussian Splats for aesthetic viewing. I seldom hear about these things in the US though

Compact NGP was my last project at NVIDIA. It's been a wild 3.5 years (almost 5 incl. internships) and I already miss my truly amazing coworkers- but here's to fun new exciting things in 2024. 😄 (now I need to buy GPUs 😭)

VideoCAD: dataset of screen recordings of OnShape models being created on UI to train "browser agents" (colloquial term) for CAD. Feels like the next year ish will be a battle of "just use existing UIs for humans" vs. "make LLM-native connectors / scaffolding"

Cad-MLLM 🗿 Augments the DeepCAD dataset by including chamfered data (chamfers not preserved though 🥺) and adding extrusion subtrees as separate data points. Vicuna-7B with Michaelangelo point cloud encoder for multimodal embedding for text + point cloud -> CAD

Martian World Models

Sekai: a video dataset for world exploration Random childhood dream I always had was to make a simulator of the entire earth. During COVID I spent a lot of time doomwatching "Walking in X" videos . Making that into a dataset seems like a fun step towards that...

アメリカのコンピュータサイエンス学部生の数、凄い勢いで伸びている。いくら需要があってもこのままだと供給が追いつきそうな雰囲気…

We had a blast at #SIGGRAPH2022 last week demoing our new modular open source #neuralfields NeRF library, Kaolin Wisp. In case you missed the session... the video recording of the overview talk and the Wisp tutorial from @OrPerel is available online! nvidia.com/en-us/on-demand/s…

今年は起業したりYCに採択されてSFに引っ越したり仕事で大変な事が本当に色々あったけれども話のネタとして1番面白いのは渋谷でアイドル(STU48)にインタビューされた事かもしれない

Generating 3D meshes of buildings from satellite imagery + coarse geometry.... using Cities: Skylines (the game) as the dataset

Neural LOD has been accepted to CVPR 2021 as an oral presentation!! 🥳

If you ever visit Japan, don't go to all the generic places on the tour guides. Go to Shimanami Kaido, a bike path where you can bike across the numerous islands of the Setouchi through giant bridges. Marvel in the beauty of the built world.

彼が巷で噂の最強京大生か🤔 wantedly.com/companies/compa…

ブラウン大学・MIT・グーグル・フェイスブック・TUM等の研究者さん達と共同で書いた”ニューラル場の手法と応用”のサーベイ論文がArXivに出ました！ ついでに”ニューラル場”の解説とサーベイに至る経歴をブログ記事として日本語でも書いたので良ければご一緒に読んで下さい！ yongyuanxi.medium.com/%E3%83…

ニューラル場（入力として座標が使われるニューラルネット）の最適化において、座標にノイズを注入するだけで最適化が早くなる事を示す論文。物理シミュレーションのPINNとかでもそのまま簡単に実証できそう

Parsing CAD drawings is one thing but parsing architectural plans takes the craziness up a notch

Over the years I've been collecting papers on "Compression for 3D" (broadly defined) on GitHub which helps for making related works. Maybe it would be useful for you too & it's likely missing lots of exciting works so PRs are always welcome 😁 github.com/tovacinni/awesome…

CAD-Assistant: AutoGen agent with CAD tools The most interesting result IMO is the table comparing different serialization *and* parameterization formats of CAD data- JSON with point-based representation works best. Serialization formats matter a lot for agents!

CVPR'19に採択された論文のデータを纏めている。取り敢えず研究機関ごとの論文数トップ３０（複数人著者がいる論文は機関ごとに一度だけカウントされる）：

🌿 Thyme: an open source dataset & model on training visual reasoning models that leverage image manipulation tools (like cropping, zooming, image processing)to aid the reasoning process

Good reminder of the aha moment I got when I read CycleGAN (also scary how long ago CycleGAN is now!)

Automatically determining machining (milling) sequences from only the final part geometry, with a transformer AI model

We had a huge showing (500+ people!) for our SIGGRAPH course on Neural Fields. Thanks to everyone who came despite the early morning slot and also s/o to my awesome co-instructors @jtompkin @psyth91 @alexyu00 😄

We've spent a lot of thought about why the "neural" part of "neural fields" matters at all. It seems to boil down to 3 things: 1. Compression 2. Inductive biases that makes fitting ('learning') easy 3. Flexible input space -> creative applications at low cost

アメリカの大学生がインターンも就活もしないで夜遅くまで勉強しているという謎のイメージは一体何処から来るのだろうか。むしろアメリカの学生の方がインターンも就活も積極的にしています。

CAD-Coder Yet another work that demonstrates better LLM-generation of CAD through conversion into a more LLM-readable format (Python scripts to generate DXF). Fine-tunes DeepSeek-R1-Distill-Llama-8B on a dataset of DXF Python scripts

共著論文がCV系のトップ国際学会（ECCV）に採択されました！SFU、UBC、DeepMindなどとの共同研究です。 流体などにおけるオイラー法（格子）とラグランジュ法（点群）の強みを組み合わせたニューラル場を使うことで三次元モデルをコンパクトに表現できます。

今、CV研究界隈では微分可能ボリュームレンダリングとニューラル場を使った三次元GANがアツい！ matthew-a-chan.github.io/EG3… yudeng.github.io/GRAM/

PDF図面を画像処理とVLMで解析して別システム（ERP、検索、LLMなど）に組み込むためのプラットフォーム

CAD-Coder (no it's not the same one as last week, yet another paper with the same name) VLM trained on a dataset of CadQuery Python code to directly generate CAD from images. What's interesting in this paper is that Qwen2.5 VL actually does pretty well as is.

PhD application season is coming up, so here's a #LumaFlythroughs of the DGP Lab at University of Toronto - a massive lab space that unites numerous students and faculty from graphics, vision, imaging, HCI, and more. My time here has been spectacular- reach out if you have Qs!

今日からまたNVIDIA社で働き始めました。自動運転車開発、CV研究、CG研究と初出社を重ねる毎に分野もグループも変わりますが今回も引き続き頑張っていきたいと思います！

Jupyter NotebookをCLIからスクリプトとして走らせて、コンフィグ管理コードを書かなくてもCLIからパラメーター管理できて、再現可能なコンフィグと変換されたJupyter Notebookが実験毎のフォルダに出力されて、グリッドサーチまでCLIからできるツールを開発しました！！！ github.com/haipera/haipera

Training industrial vision models with synthetically generated data (research by Accenture)

CADReview: Dataset of 1500 OpenSCAD programs collected from the internet, augmented with manually created errors and "feedback" to fix the errors. The actual building blocks for CAD code editing agent uses a couple of different fine-tuned models (based on LLaVA-OV 7B)

.@NvidiaAIで取り組んでいた論文が公表されました！ バウンダリを形のプロキシとして学習し、セグメンテーションとバウンダリに双方向のlossを課す事でscale-invarianceやboundary alignmentを改善し、セグメンテーションでもDeeplab v3+を越えてSOTAを達成する手法です。😆 nv-tlabs.github.io/GSCNN/

From: Learning to See Inside Opaque Liquid Containers using Speckle Vibrometry arxiv.org/abs/2507.20757

NeRF Wars III: Revenge of the Triangle arxiv.org/abs/2311.05607

overheard at CVPR: “no one codes in cuda anymore” 🤨

#つぶやきGLSL void main(){vec2 p=gl_FragCoord.xy;vec4 k=vec4(1,.3,.8,0);vec2 n=5*(p+p-r)/r.y;n*=2/(.5+n.y);n.y-=t*2;n=.05/sqrt(abs(fract(n*.4)-.5));c+=vec4(1,.2,0,0)*-(length(p-r/2)-r.y/4)*.01+p.y/r.y*k*.8;if(p.y>r.y/2.2)c=(n.x+n.y)*k+vec4(0,0,.1,0);c+=sin(p.y/2.+sin(t)*60)*.02;}

とりあえず起業している、もしくは起業しようと思っている方はYCの秋バッチに応募しましょう！日本からの応募数が圧倒的に少ないらしいです！！

you can just manufacture a NeRF (in solid resin)

I think a lot about lines these days (in the form of computer vision on CAD drawings to count and measure things). It's probably a curse since my first ever paper in computer vision was also about detecting lines. He who rules the line rules the world 📐😎📏

GeoCAD: selective editing of CAD - based on the FlexCAD text format and a selected sketch loop. The instruction fine-tuning process is interesting since it's based on masking out a sketch loop and making it predict the sketch shape based on a natural language description

Cadmium 🌊 - Ansys paper on synthetic labeling of DeepCAD dataset with GPT 4.1 generated annotations from the ground truth + multi view images to train a better text-to-CAD model with Qwen2.5-Coder . Doesn't actually outperform human labeling (Text2CAD) - but gets close!

ちなみに二次元の画像ではなく「人間の３D幾何形状」を直に生成する最新技術もあります！ （前職で少しだけ手伝わせてもらった論文です） research.nvidia.com/labs/nxp…

DPO RL fine-tuning for image-to-CAD generation, diffusion transformer for generation and CAD code compiler as reward model (curious why not fine-tune an LLM- maybe just being able to train on a single RTX6000?)

CReFT-CAD: GOT-OCR2.0 reinforcement fine-tuned on parameter prediction tasks from orthographically projected images of CAD. So much of "CAD" data is actually just in PDFs- so super important line of work that explores 3D reasoning from orthographic 2D.

リクルート財団の奨学金受けてみようかと思ったけど、応募条件が”分野の世界トップ２０大学に通っていること”な癖に”推薦所が外国語な場合日本語に訳せ”って一体どういうことなの… 推薦状はそもそも生徒に見せるものじゃないしそもそも”グローバル”は一体どこに…

Gaussian Splat＋三次元生成の論文が一つどころか二つ同時に出ているのが戦国時代感をさらに醸し出している。コードが両方すでにオープンソース公開されているのが素晴らしい！ gsgen3d.github.io

AIで自分を３D化してみたらつむじはげにされた