Olivia
Online
Mohammadjavad (Javad) Maheronnaghsh
@j_maheronnaghsh
AI/ML Engineer | Efficient ML . Computer Vision . GenAI | Graduated @ UofT × Meta
Toronto, Ontario
Joined November 2025
146 Following
107 Followers
496 Posts
Pinned Tweet
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
Need 31 more to reach 100. If you are into AI, Agents, LLM, vision: let's connect !
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDIb49
Post: https://t.co/YT8UqrLSmK
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDIb49
Post: https://t.co/YT8UqrLSmK
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDHDeB
Post: https://t.co/YT8UqrLkxc
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDIb49
Post: https://t.co/YT8UqrLSmK
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDIb49
Post: https://t.co/YT8UqrLSmK
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
🚀 Check out our latest work with #Meta and #UofT on accelerating XR predistortion with efficient deep learning.
106× fewer parameters and 6× fewer FLOPs while maintaining strong image quality.
Paper: https://t.co/iqntLDIb49
Post: https://t.co/YT8UqrLSmK
#AI #MachineLearning #ComputerVision #XR
XR headsets and glasses (AR, VR and MR) have a strange problem. The optics that put the picture in front of your eyes also bend and tint it.
The usual fix is to pre-distort every frame in the opposite direction, so it comes out right after it passes through the lens. A big image-to-image network can do that well. It's also far too heavy for a headset running on a battery.
We did this work with Meta, so we designed around the limits of a real wearable: a power budget of about 2 W and almost no room for extra latency.
Our preprint on Spatons is now on arXiv:
https://t.co/iqntLDHDeB
A Spaton is a tiny neural network, a few thousand parameters, that corrects one pixel at a time using only a small patch around it. We chain two of them. The first handles geometry. The second looks at a 3×3 neighbourhood and nudges colour and brightness back to where they should be.
On our hardest test, with ten distortions stacked on top of each other, that two-block pipeline matched CompenNet++ in SSIM (0.944 vs 0.944) with 106× fewer parameters and 6× fewer FLOPs.
The geometry block only cares about pixel position, so once it's trained we can replace it with a lookup table. On CPUs that made the geometry stage 13 to 104× faster with no loss in quality.
The short video below shows how it all fits together.
This is joint work between the @UofT, the @VectorInst and @Meta, together with Sushant Kondguli, Steve Mann and Andreas Moshovos.
There's still a lot left to do. Colour fringing from chromatic dispersion is where we trail the big models. Our per-pixel colour stage also doesn't gain as much from higher resolutions as a convolutional network does, and a wider, multi-scale colour stage should help with both.
So far we've tested on synthetic distortions plus one real projector-camera dataset. The big next step is running Spatons through real headset optics, where sensor noise and drift between devices will show up. For distortions with no clean formula, we also need a better learned model of the optics to train against. A SystemVerilog version of the pipeline is underway too, so we can measure real energy use instead of estimating it from FLOPs.
And I'm curious what breaks first.
Trends for you
Most Popular Users
Elon Musk
@elonmusk
241.5M followers
Barack Obama
@barackobama
118.8M followers
Cristiano Ronaldo
@cristiano
114.5M followers
Donald J. Trump
@realdonaldtrump
111.9M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.3M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90M followers
Taylor Swift
@taylorswift13
84M followers
Lady Gaga
@ladygaga
75.5M followers
Virat Kohli
@imvkohli
73.5M followers
Kim Kardashian
@kimkardashian
70.9M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66.5M followers
Bill Gates
@billgates
65.3M followers
Selena Gomez
@selenagomez
63.1M followers
The Ellen Show
@theellenshow
62.2M followers
CNN
@cnn
61.7M followers
X
@x
60.7M followers