How can we help you today?

Start a new topic
Discussions Cordova, Titanium, Unity, Xamarin Unity Questions or Problems

Get target image from camera in Unity

Get target image from camera in Unity
1 person has this problem
24 CommentsSorted by Oldest First

Hi, in Unity I want to get camera's feed from target and then process it. What I need is to be able to recognize when the predefined target is in camera's field of view and then crop and distort the part of the feed where the target is and then process the target image for a project I'm working on.
I'll appreciate if you can help with this.

Thanks

Hi, 

The CanvasCameraImage GameObject has a RawImage component that during runtime will reference a texture that contains the camera feed. You should be able to use that. Keep in mind that this behaviour will change in the future, but we will make sure that there will still be a way to access the camera frame.

Best regards,
Alexandru

Thanks very much Alexandru
Is there any way to extract and crop the target pixels from this feed ?  What I'm trying to do is 1-get camera feed 2-know when the target is in the field of view of the camera 3-extract and crop the target pixels from that feed 4-de-warp the pixels of the target
I wanted to ask if there are any ways to do this in Wikitude
Thanks very much

Best Regards

Hi, 

As mentioned earlier, you can get the camera feed from CanvasCameraImage. After that, you can get the actual pixels of the texture using the Texture2D.GetPixels function and you can apply any transformation you want. Keep in mind that GetPixels might be slow, so profile to see if the performance is acceptable for your use case.
The OnEnterFieldOfView event on the TrackableBehaviour will be triggered when a target becomes visible and from the string parameter that is passed, you figure out which target was recognized.

Best regards,
Alexandru

Hi Alexandru
What I'm trying to achieve here is to do a sample of Coloring book where you capture the target image and then wrap it as a texture to the augmented model.
Do you have any suggestions on how to achieve that with Wikitude?

Thanks

Hi,

In the latest version of the Unity Plugin, you can access the camera feed from WikitudeCamera.CameraTexture as a Texture2D. This texture is updated every frame, so if you want a snapshot of it, you will probably want to clone it by using GetPixels32() and then SetPixels32(). If you want to apply any cropping and warping, this would be the place to do it. You can then apply this clone to the Material that the augmented model is using.

Hope this helps, but if you can provide more information about the app, I'll try to provide more info.

Best regards,
Alexandru

Thanks very much Alexandru, I just got some time to work again on this problem. As you said you can get camera feed asa Texture2D, but that is very raw, and for example you can save the image when the target is recognized. But in the case of this problem you will also need to know in that image where the target is and crop it from the rest of the Texture2D. and also you'll need to warp it to undo the possible perspective in it. In Vuforia there is a script here which does all of this (Captureing camera feed, cropping image and undo its perspective) so you can easily get the Colored target image and use it as a texture for your 3d model. here's the script for Vuforia : 
https://github.com/maximrouf/RegionCapture
It would be great if there was a similar solution for Wikitude.
Thanks very much
Best Regards

Hello Alexandru,

Thank you for your constant feedback! I tried to use the GetPixels32 and SetPixels32 approach that you recommended to clone the CameraTexture, but unfortunately it doesn't work. It seems like the CameraTexture is not readable. It's possible to apply the CameraTexture directly to an object so that there's a reference to it and we can see the live feed from the camera, but when I try to create a copy/snapshot via GetPixels32, SetPixels32 and Apply the texture stays empty.

Could you please provide a simple example that works?

Thank you very much in advance!
Chris

Thanks Christiaan. So we have our problem at step 1. I havne't tested this in action with Wikitude, though I have implemented a sample of this in Vuforia using the RegionCapture script. It would be great if we could do this in Wikitude too.
Thanks again very much
Best Regards

Hi,

Here is a quick script I made as proof of concept of how this problem can be approached. It relies on dummy transforms attached to the corners of the targets, but these can also be generated automatically if you know the aspect ratio of the target you are tracking. This information is not currently available in the Unity API, but we will see if it can be added in the future. Until then you need to do a bit of bookeeping for this. When the image is tracked, the positions of the corners are transformed into screenspace coordinates. I then update a quad with UVs created from these coordinates and display it. The script needs to respond to OnEnter and OnExitFieldOfVision events from the Trackable to work correctly.

There are some downsides to this approach. The mesh update has a bit of lag so the cropped image is a bit behind the feed and there is some warping at extreme angles that could be improved with a more dense mesh, but I hope this will be a useful starting point. If you have any questions, please let me know.

Best regards,
Alexandru
1 person likes this

Hi Alexandru,
Thanks very much. I wanted to get back to this project and work on it a bit more but I couldn't this time download the script you mention (maybe because of forum update).

I'll appreciate if you can kindly upload it again.

Thanks very much
Best Regards

Hi Alexandru,
Thanks to your instructions I could reach what I had in mind , but the problem is I can't do it with Wikitude.

The problem is I couldn't get Wikitude Camera feed. I tried WikitudeCamera.CameraTexture and also tried to use RenderTexture. But none worked for me and I get a blank Texture.

I'll appreciate if you can help me with this.

Thanks very much
Best Regards

I've been having the same problem as @m0j1 42420. WikitudeCamera.CameraTexture is the right size but is just white when copied to a rawImage.

So after a lot of digging (on Android), I have a hunch that WikitudeCamera.CameraTexture is not a 'proper' Texture2D or it's in the wrong format or something else entirely. I was able to render the video feed to a rawImage on the UI by casting WikitudeCamera.CameraTexture to a Texture. (eg rawImage.texture = (Texture)WikitudeCamera.CameraTexture;) which is what Wikitude do with the BackgroundCamera that renders the feed to the screen.


I don't know why this is the case and would hope Wikitude will be able to shed some late on this.

Login or Signup to post a comment
More topics in Unity Questions or Problems
See all 594 topics
© 2021 Wikitude, a Qualcomm company