Start a new topic

Get target image from camera in Unity

Get target image from camera in Unity

1 person has this problem


Hi,

Here is a quick script I made as proof of concept of how this problem can be approached. It relies on dummy transforms attached to the corners of the targets, but these can also be generated automatically if you know the aspect ratio of the target you are tracking. This information is not currently available in the Unity API, but we will see if it can be added in the future. Until then you need to do a bit of bookeeping for this. When the image is tracked, the positions of the corners are transformed into screenspace coordinates. I then update a quad with UVs created from these coordinates and display it. The script needs to respond to OnEnter and OnExitFieldOfVision events from the Trackable to work correctly.

There are some downsides to this approach. The mesh update has a bit of lag so the cropped image is a bit behind the feed and there is some warping at extreme angles that could be improved with a more dense mesh, but I hope this will be a useful starting point. If you have any questions, please let me know.

Best regards,
Alexandru

1 person likes this

I've been having the same problem as @m0j1 42420. WikitudeCamera.CameraTexture is the right size but is just white when copied to a rawImage.

Hi Alexandru,
Thanks very much. I wanted to get back to this project and work on it a bit more but I couldn't this time download the script you mention (maybe because of forum update).

I'll appreciate if you can kindly upload it again.

Thanks very much
Best Regards

So after a lot of digging (on Android), I have a hunch that WikitudeCamera.CameraTexture is not a 'proper' Texture2D or it's in the wrong format or something else entirely. I was able to render the video feed to a rawImage on the UI by casting WikitudeCamera.CameraTexture to a Texture. (eg rawImage.texture = (Texture)WikitudeCamera.CameraTexture;) which is what Wikitude do with the BackgroundCamera that renders the feed to the screen.


I don't know why this is the case and would hope Wikitude will be able to shed some late on this.


Hi, 

As mentioned earlier, you can get the camera feed from CanvasCameraImage. After that, you can get the actual pixels of the texture using the Texture2D.GetPixels function and you can apply any transformation you want. Keep in mind that GetPixels might be slow, so profile to see if the performance is acceptable for your use case.
The OnEnterFieldOfView event on the TrackableBehaviour will be triggered when a target becomes visible and from the string parameter that is passed, you figure out which target was recognized.

Best regards,
Alexandru

In case anybody is trying to do this, the earlier method doesn't work -- at least on iOS, wikitudecamera.cameraTexture doesn't get updated unless the BackgroundCamera's camera is present and active (as soon as you disable the camera, cameraTexture no longer updates). So I had to just send that camera backwards in depth so I could draw over it. This works differently in the editor, which made this a huge pain to debug. 


And I was only able to do that because of the comment above that happens to mention the hidden gameobject. Not sure why that's hidden -- only makes working with wikitude harder. 

Hi! is there a way to render the camera texture to a RawImage without Wikitude also rendering the camera feed in the background?

 

I can render to a custom image just fine with rawImage.texture = wikitudeCamera.CameraTexture; but only as long as the WikitudeCamera component has Enable Camera Rendering set to true in the Editor, which makes it render the texture to the background as well. If I set it to false I get no background but also no texture (white image) to render to the RawImage.

 

Is this even possible (outside of covering the background with something or doing ugly hacks)?

 

Thanks!

Hi Alexandru,
Thanks to your instructions I could reach what I had in mind , but the problem is I can't do it with Wikitude.

The problem is I couldn't get Wikitude Camera feed. I tried WikitudeCamera.CameraTexture and also tried to use RenderTexture. But none worked for me and I get a blank Texture.

I'll appreciate if you can help me with this.

Thanks very much
Best Regards


Thanks very much Alexandru, I just got some time to work again on this problem. As you said you can get camera feed asa Texture2D, but that is very raw, and for example you can save the image when the target is recognized. But in the case of this problem you will also need to know in that image where the target is and crop it from the rest of the Texture2D. and also you'll need to warp it to undo the possible perspective in it. In Vuforia there is a script here which does all of this (Captureing camera feed, cropping image and undo its perspective) so you can easily get the Colored target image and use it as a texture for your 3d model. here's the script for Vuforia : 
https://github.com/maximrouf/RegionCapture
It would be great if there was a similar solution for Wikitude.
Thanks very much
Best Regards

Hi.


I apologize for the late reply.


Here is a snippet of code that captures the frame and saves it on disk

 

using UnityEngine;
using System.Collections;
using Wikitude;
using System;
using System.IO;

public class ScreenshotCapture : MonoBehaviour {

	private WikitudeCamera _wikitudeCamera;

	private RenderTexture _targetRenderTexture;
	private Texture2D _cameraFeed;

	void Start () {
		_wikitudeCamera = FindObjectOfType<WikitudeCamera>();
		if (_wikitudeCamera == null) {
			Debug.LogError("No WikitudeCamera found!");
			Destroy(this);
		} else {
			CreateTextures();
		}
	}
	
	private void CreateTextures() {
		_targetRenderTexture = new RenderTexture (Screen.width, Screen.height, 24);
		_cameraFeed = new Texture2D (Screen.width, Screen.height);
	}

	void Update () {
		if (Input.GetKey(KeyCode.Space) || Input.GetMouseButton(0)) {
			string path = Path.Combine(Application.persistentDataPath, "wikitudeFeedImg_" + System.DateTime.Now.ToString("hh_mm_ss") + ".png");
			Debug.Log("path: " + path);
			StartCoroutine(CaptureScreenshot(_wikitudeCamera, path));
		}
	}

	private IEnumerator CaptureScreenshot(WikitudeCamera wikitudeCamera_, string path_) 		{
		yield return new WaitForEndOfFrame ();
		
		if (_targetRenderTexture.width != Screen.width || _targetRenderTexture.height != Screen.height) {
			CreateTextures();
		}

		Graphics.Blit(wikitudeCamera_.CameraTexture, _targetRenderTexture);
		RenderTexture.active = _targetRenderTexture;

		_cameraFeed.ReadPixels(new Rect(0, 0, _cameraFeed.width, _cameraFeed.height),0,0);
		_cameraFeed.Apply();

		byte[] bytes;
		bytes = _cameraFeed.EncodeToPNG();
		File.WriteAllBytes(path_, bytes);
	}
}

 


Best regards,

Alexandru.


Hello Alexandru,

Thank you for your constant feedback! I tried to use the GetPixels32 and SetPixels32 approach that you recommended to clone the CameraTexture, but unfortunately it doesn't work. It seems like the CameraTexture is not readable. It's possible to apply the CameraTexture directly to an object so that there's a reference to it and we can see the live feed from the camera, but when I try to create a copy/snapshot via GetPixels32, SetPixels32 and Apply the texture stays empty.

Could you please provide a simple example that works?

Thank you very much in advance!
Chris

Hi, 


There currently isn't a proper way to do this.


However, the background is rendered by a hidden object named "BackgroundCamera", so you could simply destroy it and rendering will also stop. This object is created during the Start method of the WikitudeCamera, so make sure to wait for that before searching for the BackgroundCamera. Also make sure to enable clearFlags on your main camera, as they are set to DepthOnly by default.


Here is a short snippet of how this can be done:


IEnumerator Start() {

    yield return null;

    Destroy(GameObject.Find("BackgroundCamera"));

    Camera.main.clearFlags = CameraClearFlags.Color;

}


Best regards,

Alexandru

The link just takes me back to the forum home?

My mistake. 

I updated the previous post.


Best regards,

Alexandru

Login or Signup to post a comment