iOS Question MLKit Selfie Segmentation

[walterf25]

I was able to use the MLKit subject segmentation code in an Android Application which successfully removes the entire background from an image leaving only the subject in the image.

I have been trying to do the same thing for B4i, but have not had much success, I saw this post here from Erel, but this seems to remove the subject rather than the background, I have been trying for about a week now to find a solution and have tried different inline objective c modificatons, but have not been able to achieve my goal.

Most of the modifications I have tried are suggestions from ChatGPT and code I've found in other places,

Here's one example

-(void (^)(id, id)) createBlock:(CVPixelBufferRef)pixelBuffer {
    void (^block)(id result, id error) = ^void (MLKSegmentationMask* mask, id error) {
        if (error != nil || mask == nil) {
            [B4I shared].lastError = error;
            [self.bi raiseUIEvent:nil event:@"process_result::" params:@[@(false), [NSNull null]]];
        } else {
            // Get mask dimensions and pixel buffer
            CVPixelBufferRef maskPixelBuffer = mask.buffer;
            CIImage *ciMaskImage = [CIImage imageWithCVPixelBuffer:maskPixelBuffer];
           
            // Ensure mask is in grayscale
            ciMaskImage = [ciMaskImage imageByApplyingFilter:@"CIColorMatrix"
                                   withInputParameters:@{
                                       @"inputRVector": [CIVector vectorWithX:0 Y:0 Z:0 W:1],
                                       @"inputGVector": [CIVector vectorWithX:0 Y:0 Z:0 W:1],
                                       @"inputBVector": [CIVector vectorWithX:0 Y:0 Z:0 W:1],
                                       @"inputAVector": [CIVector vectorWithX:0 Y:0 Z:0 W:1]
                                   }];
           
            CIContext *temporaryContext = [CIContext contextWithOptions:nil];
           
            // Convert the original pixel buffer to CIImage
            CIImage *originalImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];

            // Ensure original image and mask have matching dimensions
            if (CVPixelBufferGetWidth(maskPixelBuffer) != CVPixelBufferGetWidth(pixelBuffer) ||
                CVPixelBufferGetHeight(maskPixelBuffer) != CVPixelBufferGetHeight(pixelBuffer)) {
                // Resize original image to match mask dimensions if necessary
                originalImage = [originalImage imageByApplyingFilter:@"CILanczosScaleTransform"
                                                withInputParameters:@{
                                                    kCIInputScaleKey: @(CVPixelBufferGetWidth(maskPixelBuffer) / CVPixelBufferGetWidth(pixelBuffer)),
                                                    kCIInputAspectRatioKey: @(1.0)
                                                }];
            }

            // Blend the mask and the original image
            CIImage *maskedImage = [originalImage imageByApplyingFilter:@"CIBlendWithAlphaMask"
                                    withInputParameters:@{
                                        kCIInputMaskImageKey: ciMaskImage
                                    }];
           
            // Convert the result to a UIImage
            CGImageRef videoImage = [temporaryContext createCGImage:maskedImage
                    fromRect:CGRectMake(0, 0,
                    CVPixelBufferGetWidth(maskPixelBuffer),
                    CVPixelBufferGetHeight(maskPixelBuffer))];
                   
            UIImage *uiImage = [UIImage imageWithCGImage:videoImage];
            CGImageRelease(videoImage);
           
            // Raise event with the processed image
            [self.bi raiseUIEvent:nil event:@"process_result::" params:@[@(true), uiImage]];
        }
    };
    return block;
}

This code returns an image that is about 95% blank with some colored streaks going across the image, so obviously it doesn't work, Has anyone been able to remove the background from an image leaving only the subject, I realize that the library being used in this case is selfie segmentation and not subject segmentation, it seems subject segmentation only works on Android? I can not find any documentation from MLKit for iOS only selfie segmentation which I realize may be totally different.

Still I would like to find a solution, if anyone has any ideas or thoughts please let me know.

Regards,
Walter

 

[Erel]

Private Sub HandeChooserResult (ChooserResult As MediaChooserResult)
    For Each v As B4XView In Root.GetAllViewsRecursive
        If v.Tag Is B4XImageView Then v.Tag.As(B4XImageView).Clear
    Next
    If ChooserResult.Success = False Then Return
    Dim bmp As B4XBitmap = xui.LoadBitmap(ChooserResult.MediaDir, ChooserResult.MediaFile)
    
    If bmp.Width > 2000 Or bmp.Height > 2000 Then bmp = bmp.Resize(2000, 2000, True)
    #if B4A
    If ChooserResult.Mime = "image/jpeg" Then
        bmp = RotateJpegs(ChooserResult, bmp)
    End If
    #end if
    
    ivOrigSmall.Bitmap = bmp
    ivOriginal.mBackgroundColor = xui.Color_Transparent
    Wait For (SelfieSegmenter.Process(bmp)) Complete (res As SelfieSegmentationResult)
    If res.Success Then
        Dim AfterMask As B4XBitmap = RemoveMask(bmp, res.ForegroundBitmap)
        ivOriginal.Bitmap = AfterMask
        ivMaskSmall.Bitmap = res.ForegroundBitmap
    End If
End Sub

Private Sub RemoveMask(Selfie As B4XBitmap, Mask As B4XBitmap) As B4XBitmap
    Dim SelfieBC As BitmapCreator = CreateBC(Selfie)
    Dim MaskBC As BitmapCreator = CreateBC(Mask)
    Dim oldargb, newargb, a As ARGBColor
    SelfieBC.ColorToARGB(xui.Color_Black, oldargb)
    SelfieBC.ColorToARGB(xui.Color_Transparent, newargb)
    For x = 0 To SelfieBC.mWidth - 1
        For y = 0 To SelfieBC.mHeight - 1
            MaskBC.GetARGB(x, y, a)
            If a.r = oldargb.r And a.g = oldargb.g And a.b = oldargb.b Then
                SelfieBC.SetARGB(x, y, newargb)
            End If
        Next
    Next
    Return SelfieBC.Bitmap
End Sub

Tested in B4i only. It might need a different color in B4A.

Note that if you change anything in RemoveMask then you will need to clean project (ctrl + p) or it will run very slowly.

Depends on BitmapCreator.