B4J Question [B4X]How to compress and decompress strings using gzip.

[byz]

Base64 + Gzip is a widely adopted method for efficient and text-safe data transmission in web APIs and network protocols. the WEBAPI interface compresses a large json string through gzip and returns it as a base64 encoded string. Therefore it needs to be unzipped. I've done it in vb.net, but b4x always fails.

Sub DecompressString( base64 As String)
    Dim compress As CompressedStreams
    Dim su As StringUtils
    Dim compressed(), decompressed() As Byte
    compressed = su.DecodeBase64(base64)
    decompressed= compress.DecompressBytes(compressed, "gzip")
    xui.MsgboxAsync(BytesToString(decompressed,0, decompressed.Length, "UTF8"), "")
End Sub

LINE 6 ERROR TIPS: Caused by: java.util.zip.ZipException: Not in GZIP format

It's not clear if it's my method that's wrong or if it's a compatibility issue.
Attached are the test items
================A complete solution===================
This is the correct compression and compression code that supports gzip and zlib compression algorithms. In the second attachment is an OK test item.

[B4X]How to compress and decompress strings using gzip.

Base64 + Gzip is a widely adopted method for efficient and text-safe data transmission in web APIs and network protocols. the WEBAPI interface compresses a large json string through gzip and returns it as a base64 encoded string. Therefore it needs to be unzipped. I've done it in vb.net, but b4x...

www.b4x.com

 

[aeric]

I found this:

GZIP: B4x & PHP via okHttpUtils

Often transfered data (esp. JSON fromatted) get's huge. This is about to GZIP encode in PHP and decode in B4X. Like every data manipulation (encryption, zipping) it is BYTE based. So the main thing is to convert it between bytes and string format. Libs: jRandomAccessFile PHP: GZENCODE needs a...

www.b4x.com

 

[byz]

I tried, but it didn't work, he was returning a direct stream from httprespose, not base64

 

[teddybear]

Sub DecompressString( base64 As String)
     ...
    compressed = su.DecodeBase64(base64)
    decompressed= compress.DecompressBytes(compressed, "gzip")
    ...
End Sub

LINE 6 ERROR TIPS: Caused by: java.util.zip.ZipException: Not in GZIP format

    compressed = su.DecodeBase64(base64)
    decompressed= compress.DecompressBytes(compressed, "gzip")

This code is wrong, it(compressed) only is bytes which decoded Base64, not GZIP format. you can directly convert the bytes to string as below

    Dim bc As ByteConverter
    Log(bc.StringFromBytes(compressed,"utf8"))

 

[aeric]

decompressed= compress.DecompressBytes(compressed, "gzip")

use "zlib"

 

[byz]

use "zlib"

I tried, but it didn't work.Does it seem like the encoding is incompatible?

 

[aeric]

I tried, but it didn't work.Does it seem like the encoding is incompatible?

Show your compress code. It is reversed of decompress.

 

[byz]

Show your compress code. It is reversed of decompress.

It is possible to use the forum's decompression and compression code. But what is returned from the server is a gzip string in base64 format. Cannot be correctly decompressed by b4x. I have included the test files and project in the attached project.

 

[aeric]

I found that zlib+base64 works but not works for gzip+base64.

 

[emexes]

Grok says it is Base64 and Gzip, but not text, and thus interpreting the decompressed bytes as UTF-8 is going to fail bad.

https://x.com/i/grok/share/sT6FDlojExrCb9zCBTH5GxURE

Note that I didn't even mention Gzip to Grok: it independently identified that it was compressed, and how.

 

[emexes]

The first three bytes of the Base64 decoding ie 1F 8B 08:

match the description of Gzip format:

https://en.wikipedia.org/wiki/Gzip#:~:text="gzip" is often also used,and the operating system ID.

so that bodes well. Again, if it is not UTF-8 text, then trying to decode it as such is going to fail.

 

[aeric]

Check the related results:
https://www.b4x.com/android/forum/pages/results/?query=base64+decode+gzip

https://www.b4x.com/android/forum/t...-b4a-decompression-problem.72929/#post-463498

if you are using a web server as a back end to retrieve information, then, with most modern web servers, the text data is compressed out of the box

 

[emexes]

I've done it in vb.net, but b4x always fails.

I missed this bit. Let me have a closer look at it as UTF-8. Although the leading zeroes look un-texty and thus un-JSONy too. The could be null-terminators for lines but, if so: why 7 empty lines up front, and then (by eye) no line terminators after that? On the other hand, I've seen JSON scrunched onto a single line, so perhaps that's plausible. I'd expect a line terminator at the end of that single line, though.

 

[emexes]

closer look at it as UTF-8

It doesn't look like UTF-8 because in UTF-8, high-bit bytes are never alone, they always occur in groups of 2 to 4 (or maybe 5). So the 55 91 4d is not UTF-8 because it contains the single high-bit byte 91 by itself, without a follow-on second byte of a multibyte sequence.

Can you show us the VB code? And confirm that it processes the supplied example base64.txt file correctly, presumably recovering the supplied sample json.txt?

 

[byz]

It doesn't look like UTF-8 because in UTF-8, high-bit bytes are never alone, they always occur in groups of 2 to 4 (or maybe 5). So the 55 91 4d is not UTF-8 because it contains the single high-bit byte 91 by itself, without a follow-on second byte of a multibyte sequence.

Can you show us the VB code? And confirm that it processes the supplied example base64.txt file correctly, presumably recovering the supplied sample json.txt?

This is my VB.net code.

Imports System.IO
Imports System.IO.Compression
Imports System.Text

Public Class Gzip
    ''' <summary>
    ''' 将字符串压缩并转换为 Base64 字符串
    ''' </summary>
    ''' <param name="input">要压缩的字符串</param>
    ''' <returns>压缩后的 Base64 字符串</returns>
    Public Shared Function CompressString(input As String) As String
        ' 将字符串转换为字节数组
        Dim inputBytes As Byte() = Encoding.UTF8.GetBytes(input)

        ' 使用 Gzip 压缩
        Using memoryStream As New MemoryStream()
            Using gzipStream As New GZipStream(memoryStream, CompressionMode.Compress)
                gzipStream.Write(inputBytes, 0, inputBytes.Length)
            End Using

            ' 将压缩后的字节数组转换为 Base64 字符串
            Dim compressedBytes As Byte() = memoryStream.ToArray()
            Return Convert.ToBase64String(compressedBytes)
        End Using
    End Function

    ''' <summary>
    ''' 将 Base64 字符串解压缩为原始字符串
    ''' </summary>
    ''' <param name="compressedBase64">压缩后的 Base64 字符串</param>
    ''' <returns>解压缩后的原始字符串</returns>
    Public Shared Function DecompressString(compressedBase64 As String) As String
        ' 将 Base64 字符串转换为字节数组
        Dim compressedBytes As Byte() = Convert.FromBase64String(compressedBase64)

        ' 使用 Gzip 解压缩
        Using memoryStream As New MemoryStream(compressedBytes)
            Using gzipStream As New GZipStream(memoryStream, CompressionMode.Decompress)
                Using reader As New StreamReader(gzipStream, Encoding.UTF8)
                    Return reader.ReadToEnd()
                End Using
            End Using
        End Using
    End Function
End Class

 

[emexes]

Well, that all looks ok. Can you temporarily modify the VB line:

Dim inputBytes As Byte() = Encoding.UTF8.GetBytes(input)

to:

'''Dim inputBytes As Byte() = Encoding.UTF8.GetBytes(input)
Dim inputBytes As Byte() = Encoding.UTF8.GetBytes("Hello, World!")

and post the resultant Base64 output? (which may well be longer than the original 13 bytes)

 

[byz]

Well, that all looks ok. Can you temporarily modify the VB line:



to:

'''Dim inputBytes As Byte() = Encoding.UTF8.GetBytes(input)
Dim inputBytes As Byte() = Encoding.UTF8.GetBytes("Hello, World!")

and post the resultant Base64 output? (which may well be longer than the original 13 bytes)

There is test data in the attachment

 

[emexes]

Ok, something weird is going on. StringUtils.DecodeBase64 seems to also do the decompression.

 

[emexes]

StringUtils.DecodeBase64 seems to also do the decompression.

It will automatically gunzip the data when it detects the gzip header. This is true in B4J and B4A.

It'd be nice if this "feature" was mentioned in the online documentation and ide popup help:

https://www.b4x.com/b4j/help/jstringutils.html#stringutils_decodebase64

 

[William Lancee]

Your test data, based on what @emexes found
@teddybear already had the solution in #4!

    Dim su As StringUtils
    Dim s As String=File.ReadString(File.DirAssets,"base64.txt")
    Dim d2() As Byte
    d2=su.DecodeBase64(s)
    Dim bc As ByteConverter
    Log(bc.StringFromBytes(d2 , "UTF8"))
    '{"appName":"WeatherTracker","version":"2.3.5","features":["real-time updates","5-day forecast","severe weather alerts","custom locations"],"settings":{"temperatureUnit":"Celsius","notificationEnabled":true,"refreshInterval":30,"theme":"dark"},"user":{"id":"WTU-7845-2023","premiumMember":false,"lastActive":"2023-11-15T08:23:17Z"},"systemInfo":{"apiCallsToday":147,"uptime":"36:14:22","memoryUsage":"64%"},"messages":[{"id":1,"text":"Welcome to WeatherTracker!","priority":"low"},{"id":2,"text":"New update available","priority":"medium"}]}