Android Question How to receive large amounts of data from php

[kostefar]

Dear All,

So I used this great multipart example example made by DonManfred to upload large amounts of data. It works very well indeed, but now I have a problem getting the data back again from the server they were uploaded to.
I built a script in PHP that will jsonencode various items; username, timestamp and also the images that are the previously mentioned large amounts of data, base64 encoded.
This works always with retrieving one image, sometimes with two images, and never with three images. That generates an Internal Server Error, so obviously there´s a limit somewhere that I´m crossing.
It all comes back as one big URL, and that´s why I base64 encoded it in the php script. This is not very efficient, so I´m looking into how I can change the code - both on b4a and php sides - to send raw bytes instead of the clumsy way I´m doing it now.

Any suggestions?

Thanks in advance!

 

[sorex]

you should fetch the link/path to the images and download them when needed or when the json has been pulled in.

base64 massively enlarges the data as it converts binary to text so you'll end up with 140+% more data transfer even when you don't even use the image in your app.
Not to mention slow decoding times.

 

[kostefar]

you should fetch the link/path to the images and download them when needed or when the json has been pulled in.

base64 massively enlarges the data as it converts binary to text so you'll end up with 140+% more data transfer even when you don't even use the image in your app.
Not to mention slow decoding times.

Ah, I forgot to mention that this is what I first considered. What I don´t like about that method is that it´ll expose the path to where they are hosted if somebody is sniffing your wifi, and make me more vulnerable to an attacker who will only need the image name to download a copy for himself. He could also grab the base64 encoded data of course, but there I´d have an option to encrypt it.
Hmm, just thnking about that I could make a temporary folder on the server each time a file is to be downloaded, into which the data will be copied and then downloaded from there. Then the temp folder will be deleted afterwards. Thanks for somehow inspiring me to think about it this way

Other suggestions are still welcome.

 

[KMatle]

Keep your Base64 encoding. No direct downloads of files via folders. On the other hand do not download so many data. As I often say : "It's a programming/design issue". Just load 10 or 20 and then the next when the user needs it. It makes no sense to build a list with thousands of entries "because we can" I use JSON structures with B64 a lot with encryption. And yes, it's getting big but it's easy to handle.

 

[sorex]

you can always pull a temporary key from the backend and send that with every request to a php file which returns the image if the key is still valid and if some random identifier matches the identifier of the image. then you don't send paths to the file itself.

options enough beside the base64 one you just need to be creative

 

[kostefar]

Keep your Base64 encoding. No direct downloads of files via folders. On the other hand do not download so many data. As I often say : "It's a programming/design issue". Just load 10 or 20 and then the next when the user needs it. It makes no sense to build a list with thousands of entries "because we can" I use JSON structures with B64 a lot with encryption. And yes, it's getting big but it's easy to handle.

Thanks KMatie,

Well, in this case the files are downloaded when needed - not because I can Do you see any advantages in the base64 approach where I´d run into problems with sizes of over approximately 5 mb + the amount of data would increase on the network, over sorens last post, or one where files and the subfolders they´re in only exist upon download? The only disadvantage I see about that is that every time this happens, the files will take up double space on the server - temporarily.

 

[kostefar]

you can always pull a temporary key from the backend and send that with every request to a php file which returns the image if the key is still valid and if some random identifier matches the identifier of the image. then you don't send paths to the file itself.

options enough beside the base64 one you just need to be creative

Cheers Soren. This is a bit what I was thinking about, but I don´t know how to set up a download of binary data without using base64.

 

[KMatle]

Well, in this case the files are downloaded when needed - not because I can Do you see any advantages in the base64 approach where I´d run into problems with sizes of over approximately 5 mb + the amount of data would increase on the network, over sorens last post, or one where files and the subfolders they´re in only exist upon download? The only disadvantage I see about that is that every time this happens, the files will take up double space on the server - temporarily.

No doubt that B64 takes a lot of space more. I store everything in a database (no huge files, B64 is perfect for me). Large files can only be images (can you reduce the quality which gives a lot of possibilities) or other files (I'm afraid to ask what it is ). So if you can't reduce/optimize you need to use folders.

By the way: I first started with folders (one for each customer). Then I came to the question how to create a user authorisation (yep. apache can do a folder login) but i found it easier to keep everything in one db. So it's a matter of taste.

 

[sorex]

Cheers Soren. This is a bit what I was thinking about, but I don´t know how to set up a download of binary data without using base64.

it's just a download like your base64 link, the only difference is that you write the datastream to disk and not the non binary text content.

on the php side you can read the jpg and output it with the right headers so that it becomes a binary download while still hiding the actual path.

 

[kostefar]

it's just a download like your base64 link, the only difference is that you write the datastream to disk and not the non binary text content.

on the php side you can read the jpg and output it with the right headers so that it becomes a binary download while still hiding the actual path.

Thanks Soren, is this the technique you´re talking about:

http://stackoverflow.com/questions/35473994/how-to-hide-download-file-path-from-user-using-php

Is there any advantages of doing it this way other than skipping the step of copying the file and the extra temporary space taken up on server side? Securitywise it´d be the same, no?

 

[kostefar]

No doubt that B64 takes a lot of space more. I store everything in a database (no huge files, B64 is perfect for me). Large files can only be images (can you reduce the quality which gives a lot of possibilities) or other files (I'm afraid to ask what it is ). So if you can't reduce/optimize you need to use folders.

By the way: I first started with folders (one for each customer). Then I came to the question how to create a user authorisation (yep. apache can do a folder login) but i found it easier to keep everything in one db. So it's a matter of taste.

In my app, people can send whatever they want to eachother. Since this could also be video, the limit of size for uploads has to be set pretty high - perhaps at 15 mb. If somebody´s got 4 video clips pending for download the next time he logs in, it will not work with the limit I run into described in the first post, of about 5 mb´s, and somebody who´s on a mobile data plan may have to pay alot more if he receives the files as base64. For the same reason, indeed it´s a good idea to perhaps reduce the sizes of images, as you suggest. Nothing that can be done with vids though - would take too long.
Why download all data at the same time? Well, that´s how it´s designed. I did it in a "deliver when needed" fashion at first, but later on could see advantages of changing this.

 

[sorex]

Is there any advantages of doing it this way other than skipping the step of copying the file and the extra temporary space taken up on server side? Securitywise it´d be the same, no?

your method causes more overhead and need for extra space.

if you only have a few hundreds of users with not too much data it will work fine. It will go out of controll when the userbase it getting huge unless you immediatly delete upon successfull retrieval but that's again extra requests to the server that you need to send.

 

[kostefar]

your method causes more overhead and need for extra space.

if you only have a few hundreds of users with not too much data it will work fine. It will go out of controll when the userbase it getting huge unless you immediatly delete upon successfull retrieval but that's again extra requests to the server that you need to send.

Thanks Sorex, that´s indeed what I thought, so I better try to get it done the way you suggest. Is the thread I linked to indeed what you´re talking about?

 

[sorex]

yes, a tweaked method of that will do.

 

[kostefar]

yes, a tweaked method of that will do.

Thanks, on my to do list!