Android Question Camera Library and faster preview

[canalrun]

Hello,
I am developing an app very similar to the CCTV example where I capture a preview frame, convert it to a JPEG, send it through a socket, and display on another device. So far so good – it works fairly well.

As Erel mentioned in the example, five frames per second seems to be the max. I did some timing and found out the preview frame rate on my Nexus 5 was about 150 ms with no processing of the preview image. This corroborates the approximately five frames per second max.

I have been searching the B4A forums and Internet for ways to possibly speed this up – 10-15 frames per second would be a major win.

I have seen several posts mention using setPreviewCallbackWithBuffer along with addCallbackBuffer instead of setPreviewCallback. The former allows one to reuse preallocated buffers.

Erel posted the source code for the Camera library (thanks Erel) http://www.b4x.com/android/forum/threads/camera-library-v2-20.23799/#post-137872. I think he uses the latter.

Question: Does anybody have experience using the preallocated buffers? Do you think it would be worth trying to modify the Camera library to allow the use of the preallocated buffers?

Thanks,
Barry.

 

[Erel]

You don't need to modify the library for that. You can use JavaObject to call setPreviewCallbackWithBuffer.

You can get a reference to the native camera object with this code:

Dim cam As JavaObject = Camera1
cam = Camera1.GetField("camera")

See this example: Accesing third party Jar with #Additionaljar and JavaObject - Picasso

 

[canalrun]

Hello,
I've spent quite a bit of time going through the JavaObject tutorial and JO library documentation. I'm trying to follow Erel's suggestion above to use JavaObject to implement setPreviewCallbackWithBuffer.

As a first step I would like to just install a new PreviewCallback and have it call my callback Instead of the default callback.

I have written a test application which is essentially the Camera with CameraEX demo application. I made some minor modifications to CameraEX.

Public Sub StartPreview
  cam.StartPreview
End Sub

Sub Camera_Preview (Data() As Byte)
  If SubExists(target, event & "_preview") Then
    CallSub2(target, event & "_preview", Data)
  End If
End Sub

Public Sub StartPreviewBuf
  Dim camjo As JavaObject = cam ' Note cam dimmed above as: Dim cam As Camera
  camjo = camjo.GetField("camera")

  Dim callback As Object = camjo.CreateEvent("android.hardware.Camera.PreviewCallback", "Callback", Null)

  camjo.RunMethod("setPreviewCallback", Array(callback))

  cam.StartPreview
End Sub

Sub Callback_Event(MethodName As String, Args() As Object) As Object
  If MethodName = "onSuccess" Then
    ToastMessageShow("Success!!!", True)
  Else If MethodName = "onError" Then
    ToastMessageShow("Error downloading image.", True)
  End If
  Return Null
End Sub

I call StartPreviewBuf intending to install the "Callback" callback replacing the callback installed by default within the Camera library. I had hoped Callback_Event would be invoked with preview frames rather than Camera_Preview. When I run, Camera_Preview, via the callback installed by the Camera library, is called with preview frames. Callback_Event is never called.

Basically I'm lost. I don't know whether my code is ridiculous or maybe the default callback is reinstalled. Can anyone offer some suggestions?

The above is only the first step. If I gain some confidence in this, the next stop will be to use setPreviewCallbackWithBuffer in which case I'll also have to provide and reuse some buffers with addCallbackBuffer.

I have attached my test project including version 2.01 of the Camera library (2.20 is the latest version. I'm hoping it's similar).

Thanks,
Barry.

 

[Erel]

There were several problems in your code.

1. Remove Camera_Preview sub.
2. Change your code to:

Public Sub StartPreviewBuf
   Dim camjo As JavaObject = cam ' Note cam dimmed above as: Dim cam As Camera
   camjo = camjo.GetField("camera")
   Dim callback As Object = camjo.CreateEvent("android.hardware.Camera.PreviewCallback", "Callback", Null)
   camjo.RunMethod("setPreviewCallbackWithBuffer", Array(callback))
   For i = 1 To 10
     Dim b(460800) As Byte
     AddCallbackBuffer(b)
   Next
   cam.StartPreview
End Sub

Sub AddCallbackBuffer(buffer() As Byte)
   Dim camjo As JavaObject = cam ' Note cam dimmed above as: Dim cam As Camera
   camjo = camjo.GetField("camera")
   camjo.RunMethod("addCallbackBuffer", Array(buffer))
End Sub

Sub Callback_Event(MethodName As String, Args() As Object) As Object
   Log("callback")
   Dim data() As Byte = args(0)
   Log(data.Length)
   'work with the data
   'and then return it to the queue
   AddCallbackBuffer(data)
   Return Null
End Sub

In the activity:

Sub Activity_Click
   Log("click")
   camEx.StartPreviewBuf
End Sub

After several failed tests I found out that you need to wait a few seconds before starting the preview or it doesn't work. So I used the click event for that.

460800 is a hardcoded value that matches width * height * bitsPerPixel / 8. You can improve this code and calculate this value at runtime.

 

[canalrun]

Thanks,
I will incorporate this into my demo and upload.

Barry.

 

[canalrun]

With Erel's contributions I was able to put together a demo that tests using setPreviewCallbackWithBuffer versus using setPreviewCallback when capturing preview images from the camera. I've attached the demo to this post.

According to many forum posts I've read on the Internet the "WithBuffer" version of this callback was said to be significantly faster than the "allocating a new preview buffer each time" version of the function. I thought it might be worth looking at to see if I can increase the preview frame rate.

The demo has three configurations:
Default – uses the current CameraEx equivalent of setPreviewCallback
PrevBuf – uses the setPreviewCallbackWithBuffer function to set the call back
Process – same as PrevBuf adding some typical processing of the preview image

The upper left image in the demo is the preview panel initialized with Camera. The upper right image is the rotated JPEG acquired in the preview call back. The upper right image works only in the third configuration.

The button beneath the images is disabled in the first configuration. In the second and third configurations it starts the preview buffer activity. This is needed because of the anomaly mentioned by Erel above.

The row of numbers shows the preview image size (note: I hardcoded a size of 640x480. On startup the log output shows a list of supported sizes – be sure 640x480 is in the list for your device), the number of milliseconds between preview call back invocations, and the number of milliseconds required for my image processing.

The seek bar allows you to adjust the Frames per Second throttle in my image processing. This seek bar is used only in the third configuration.

The demo also includes two CustomViews I created (I'm in love with CustomViews ). The row of numbers uses something I call Heading Labels. The seek bar uses a Label Seek Bar.

The results I got were very surprising. There seems to be no speed up using the "With Buffer" version. In fact, the setPreviewCallback version seemed to be slightly faster.

Here are a couple of good links I uncovered. The first link explains why the preview images seem to be rotated funny.
http://www.impulseadventure.com/photo/exif-orientation.html

The second link talks a little bit about setPreviewCallbackWithBuffer.
http://txt.arboreus.com/2012/03/02/android-setpreviewcallbackwithbuffer-buffer-in-the.html

Barry.

 

[Erel]

I'm not surprised by the performance results. The JVM (or DVM) is very efficient with memory allocations. I didn't think that this was the bottleneck.