I found a solution which also works within a class:
In the class is a panel for the background of the dialog called "pnlBackground"
Further there is a Sub "Show" which sets pnlBackground.Visible = True.
I added this code after "pnlBackground.Visible = True":
Dim r As Reflector
r.Target = pnlBackground
r.SetOnKeyListener("OnKeyPress")
r.RunMethod2("setFocusable", "True", "java.lang.boolean")
r.RunMethod2("setFocusableInTouchMode", "True", "java.lang.boolean")
pnlBackground.RequestFocus
This is the Sub "OnKeyPress":
Sub OnKeyPress(ViewTag As Object, KeyCode As Int, KeyEvent As Object) As Boolean
If KeyCode = KeyCodes.KEYCODE_BACK Then
If pnlBackground.Visible Then
pnlBackground.Visible = False
Return(True)
Else
Return(False)
End If
Else
Return(False)
End If
End Sub
Now the back key closes only the dialog.