Android Code Snippet [B4X] Audio Streaming UDP

Based on this @Erel code that simulated a Walkie Talkie, I developed a system for streaming audio between Android and Desktop devices with UDP.
You can connect with multiple devices.

I found that some routers lose a lot of UDP packets (a high percentage) and the sound is not uniform. If you notice a discontinuous and disturbed sound, it could be the router system.

You can find the version that uses the TCP protocol in this link

Note: The B4J version requires the AudioRecord library. The B4A requires the Audio library. The B4a version works better if you have headphones (avoid the return of audio) or turn off listening when you speak and vice versa ...

 

[max123]

Hi @Star-Dust , I've developed a similar project some time ago, but I use ESP8266 and ESP32 that read audio wave from microsd card and stream it over UDP to Android (B4A) or desktop devices, PC and Raspberry Pi (B4J) or another ESP8266 with external i2S audio DAC. Tried it not only in local network but even over internet from one city to other city and works pretty well. I will release the full (Arduino IDE, B4A, B4J) projects next days.

I want suggest that you can receive the UDP stream even with VLC or Netcat, with last on Linux you can pass directly the audio stream to the linux ALSA driver and then use AudioJack to route it to other applications and even use audio plugins to process it in realtime, like equalizer, reverb, delay and so... or just to record it eg. with Audacity, Ardour etc...

On linux using VLC to receive the audio stream I use the follow command on terminal that start an UDP server that listen to:
vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=44100 udp://@:8266
I've used port 8266, you can chànge it and other parameters as you need based on the trasmitter

Using Netcat the follow command pass the received audio stream to the linux ALSA driver to be routed to other applications using AudioJack or even to PulseAudio:
nc -u -p 8266 --l | play -t raw -r 44100 -b 16 -c 2 -e signed-integer -

The ESP8266 receiver as I wrote, receives the audio stream and send to external audio DAC, then the audio will go to headphones or RCA LR connected to the amplifer, as DAC I've used the Raspberry Pimoroni PHAT DAC, the ESP8266 i2S API permits max 16 bits audio (so no 24 bits), I will try next days on ESP32 that have a better i2S API and support 2 I2s ports with configurable pins and a 24 bits audio up 192Khz, I think this not a problem because I wrote an audio library for ESP8266 and ESP32, for the first tested and works well, for the last I just need to test by connecting the DAC.

My code can stream up 192Khz 16 bit Stereo audio, but the max quality I've tried successfully is 96Khz 16 bits Stereo that is a great result, I think this is a limitation of ESP8266 that read the audio from microsd and not anyway referred to the network bandwith.

In my implementation one device work as trasmitter and other as receiver, not fullduplex, an interesting solution is to use both on both side, like a walkie-talkie or network audio call, on this direction I wrote a fast hardware optimized library for SSD1331 (96x64) RGB color small 0.95 inch oled for ESP8266 and ESP32 capable to play videos with very high framerate, up 140 FPS without audio or up 24 FPS with 44100 16 bits audio (with DAC attached) and using ESP32-CAM, i2S DAC, i2S mic etc... I want to implement an UDP video call (Audio + Video) over network between two ESPs or ESP-Desktop devices.

I noticed you warn up users for some losed packets using UDP protocol, from my tests the problem was using a broadcast address to stream to all devices, using an unicastast address I noticed that (at least on my network using FritzBox router) there is a very small amount of missed packets, like 1 missed packet every 30.000 packets, not bad.

I do not use B4R, it is installed but I use very old PCs to develop, with Win XP ?? and I've problems with it, any time I compile it recompile all the code without use a temporary file, so require very long time like the first time I compile on Arduino IDE, after this my systems are 32 bits so I cannot install anyway latest releases of the IDE. Maybe when I release my C++ code for ESP8266 and ESP32 someone interested to port it to B4R.

Please try my suggestion with VLC and Netcat if you use Linux and have some time, even on Raspberry will work, and inform me if you have success with it.

Regards
Massimo from Italy

 

[embedded]

Hi @Star-Dust , I've developed a similar project some time ago, but I use ESP8266 and ESP32 that read audio wave from microsd card and stream it over UDP to Android (B4A) or desktop devices, PC and Raspberry Pi (B4J) or another ESP8266 with external i2S audio DAC. Tried it not only in local network but even over internet from one city to other city and works pretty well. I will release the full (Arduino IDE, B4A, B4J) projects next days.

I wànt suggest that you can receive the UDP stream even with VLC or Netcat, with last on Linux you can pass directly the audio stream to the linux ALSA driver and then use AudioJack to route it to other applications and even use audio plugins to process it in realtime, like equalizer, reverb, delay and so... or just to record it eg. with Audacity, Ardour etc...

On linux using VLC to receive the audio stream I use the follow command on terminal that start an UDP server that listen to:
vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=44100 udp://@:8266
I've used port 8266, you can chànge it and other parameters as you need based on the trasmitter

Using Netcat the follow command pass the received audio stream to the linux ALSA driver to be routed to other applications using AudioJack or even to PulseAudio:
nc -u -p 8266 --l | play -t raw -r 44100 -b 16 -c 2 -e signed-integer -

The ESP8266 receiver as I wrote, receives the audio stream and send to external audio DAC, then the audio will go to headphones or RCA LR connected to the amplifer, as DAC I've used the Raspberry Pimoroni PHAT DAC, the ESP8266 i2S API permits max 16 bits audio (so no 24 bits), I will try next days on ESP32 that have a better i2S API and support 2 I2s ports with configurable pins and a 24 bits audio up 192Khz, I think this not a problem because I've wrote an audio library for ESP8266 and ESP32, for the first tested and works well, for the last I just need to test by connecting the DAC.

My code can stream up 192Khz 16 bit Stereo audio, but the max quality I've tried successfully is 96Khz 16 bits Stereo that is high quality audio too, I think this is a limitation of ESP8266 that read the audio from microsd and not anyway referred to the network bandwith.

In my implementation one device work as trasmitter and other as receiver, not fullduplex, an interesting solution is to use both on both side, like a walkie-talkie or network audio call, on this direction I've wrote a fast hardware optimized library for SSD1331 (96x64) RGB color small 0.95 inch oled for ESP8266 and ESP32 capable to play videos with very high framerate, up 140 FPS without audio or up 24 FPS with 44100 16 bits audio (with DAC attached) and using ESP32-CAM, i2S DAC, i2S mic etc... I want to implement an UDP video call (Audio + Video) over network between two ESPs or ESP-Desktop devices.

I noticed you warn up users for some missed packets using UDP protocol, from my tests the problem was using a broadcast address to stream to all devices, using an unicastast address I noticed that (at least on my network using FritzBox router) there is a very small amount of missed packets, like 1 missed packet every 30.000 packets, not bad.

I do not use B4R, it is installed but I use very old PCs to develop, with Win XP ?? and I've problems with it, any time I compile it recompile all the code without use a temporary file, so require very long time like the first time I compile on Arduino IDE, after this my systems are 32 bits so I cannot install anyway latest releases of the IDE. Maybe when I release my C++ code for ESP8266 and ESP32 someone interested to port it to B4R.

Please try my suggestion with VLC and Netcat if you use Linux and have some time, even on Raspberry will work, and inform me if you have success with it.

Regards
Massimo from Italy

Please post your project in forum...it will help us... waiting for your release...With love from India

 

[Herbert32]

I noticed you warn up users for some missed packets using UDP protocol, from my tests the problem was using a broadcast address to stream to all devices, using an unicastast address I noticed that (at least on my network using FritzBox router) there is a very small amount of missed packets, like 1 missed packet every 30.000 packets, not bad.

I made similar experiences:

UDP-Broadcast Wifi/Wifi (both parties are connected over Wifi) : up to 30% Packet-loss
UDP-Broadcast Wifi/LAN (sender connected by Wifi, receiver connected by LAN) : max 1% Packet-loss
direct UDP-Transmission (Wifi/Wifi or Wifi/LAN) : max 0.01% Packet-loss

@max123 - waiting for your Project - sounds cool

 

[max123]

For users interested to my ESP32 UDP Network Video Call project here you can find my oled library in action, it is a very small 65.536 color oled, good to make an ESP based smartwatch. These video show a very old release of my library, the first one show a small video up 30 fps that is limited because ESP read consecutive JPEG images (one by one) from flash memory. Now I've developed with B4J two encoders (RawVideoBitmap and RawVideoJpeg) capable to read a series of images and pack inside a single file in a way that microcontrollers can read frames as fast possible and (as I wrote) can reach a very high framerate (up 140 fps depending on video size and format) and can play from microsd very long videos, many hours without interruption depending on microsd card size.

I will release for free files for this project when I finish to develop it, even the video encoders compatible with my oled library and my libraries on github.

Sorry, now I cannot write other infos because I don't want going off-topics of this post, I've posted it just because I managed with UDP audio streaming as original post and some users ask for a complete audio-video solution over network.

 

[max123]

There are good news users, I will continue to use this small oled because I like a lot, now I've other two of this but with a significantly smaller PCBs, the full screen including pcb is as big as a coin, I've tried to connect to ESP32-CAM and wrote a sketch that stream a camera on the oled, the video stream is not very clear (I need to optimize it) but finally with my library I've obtained a framerate of 35-36 FPS on the oled. Other good news are that after I wrote a library for this small 0.96" (96x64) color oleds with SSD1331 controller, now I've ported it for 1.5" (128x128) color oled based on SSD1351 controller, for now I use RGB565 16 bits 65k colors, but it also support RGB666 18 bits 244k colors. Apart some issues drawing on it in Portrait and Reverse Portrait I'm able to play 128x72 videos (from microsd using ESP8266 Wemos D1 Mini) @ 83 FPS but I limited at 38Mhz SPI speed because I still used 25 centimeters long wires from breadboard, tried 40Mhz but with screen artifacts, anyway I use other oleds on breadboard with short wires @ 50-60Mhz without problems, maybe 80Mhz can work directly. Next days I will try it @ 50Mhz with short wires to speed up. This oled is bigger than others but can be used to create a relatively big smartwatch too, like a TTGO T-Watch but with Oled instead of TFT, so more readable, but unfortunately without touch screen. Now that I've wrote both ESPFastSSD1331 and ESPFastSSD1351 libraries, on both oleds (using my ESPAudioClass library) I'm able to play a video of 23-24 FPS (from microsd) in sync with audio 44100 16bits Stereo out of external PCM5102A audio DAC. Apart some very small dropouts (I need to tune it) the video and audio are excellent, for now I've tried max 20 minutes of video with audio without interruptions (around 30000 frames), but tried the same video in loop for 8 hours and seem to work very well. This on ESP8266 @ 160Mhz and with 25 centimeters long wires from breadboard, and with SPI @ 38Mhz so I will suppose that with short wires and SPI @ 50-80Mhz (or on ESP32 clocked to 240Mhz) it work like a charm. I'm able to download videos from Youtube, then use FFMPEG to extract (and resize) all frames to bitmap format, extract 44100 stereo audio from video as wave file (.wav) that my audio library can read, then pack all frames inside a single video file .rvb format (with RawVideoBitmap encoder I wrote with B4J) then ESP8266 read from microsd two different files, a .wav file for audio and a .rvb for video. Next days I will post on this forum both my RawVideoBitmap and RawVideoJpeg encoders with a full code, for RawVideoBitmap it use BitmapCreator, show a realtime video preview while encode the video fille, it support RGB565, RGB66, RGB88, BlackAndWhite output formats, yes, can encode videos even in black and white to view on monocromatic SSD1306 oleds, but still not finished yet. For now only RGB565 work, the code is not fully finished. In future I want to manage with FFMPEG (and shell) directly inside my encoder so it convert directly MP4 videos and extract audio. RVB and RVJ are my proprietary video formats that embedded devices can read as fast possible to obtain a very high frame rate, the first encodes bitmap images inside a video, the second encodes jpeg compressed images inside a video. Both have an header (like wave file has) that specifies useful infos to be used while play back, like a file signature, the color format, the total number of frames, the data chunk size etc... Stay tuned!!!

 

[bjfhs]

when i test b4a udp AudioConference ,i got error: Unsupported option: --v4-signing-enabled. See --help for supported options

 

[Star-Dust]

See This

B4A 10.7 BETA#2 Error

today I installed a new update b4a to try App Bundles and it worked great but when I tried for release I occurred this error: Unsupported option: --v4-signing-enabled. See --help for supported options. appreciated any help

www.b4x.com

 

[bjfhs]

See This

B4A 10.7 BETA#2 Error

today I installed a new update b4a to try App Bundles and it worked great but when I tried for release I occurred this error: Unsupported option: --v4-signing-enabled. See --help for supported options. appreciated any help

www.b4x.com

Thank you .today,I test your code,it work well but with a long delay about 1-2S,how to reduce it?

 

[Star-Dust]

This depends on the network and the protocol.
Using "Audio Streaming TCP" and you will have different results

 

[bjfhs]

It work in LAN with UDP,but with a long delay.

 

[Star-Dust]

The delay depends on the network and the protocol UDP.
Using "Audio Streaming TCP" and you will have different results (less delay)

 

[bjfhs]

yes,I test "Audio Streaming TCP" , it work better than udp (delay about 0.3S).but the server and client in the same room,I need less delay(<0.1S).

 

[AISH@123]

Hi @Star-Dust , I've developed a similar project some time ago, but I use ESP8266 and ESP32 that read audio wave from microsd card and stream it over UDP to Android (B4A) or desktop devices, PC and Raspberry Pi (B4J) or another ESP8266 with external i2S audio DAC. Tried it not only in local network but even over internet from one city to other city and works pretty well. I will release the full (Arduino IDE, B4A, B4J) projects next days.

I want suggest that you can receive the UDP stream even with VLC or Netcat, with last on Linux you can pass directly the audio stream to the linux ALSA driver and then use AudioJack to route it to other applications and even use audio plugins to process it in realtime, like equalizer, reverb, delay and so... or just to record it eg. with Audacity, Ardour etc...

On linux using VLC to receive the audio stream I use the follow command on terminal that start an UDP server that listen to:
vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=44100 udp://@:8266
I've used port 8266, you can chànge it and other parameters as you need based on the trasmitter

Using Netcat the follow command pass the received audio stream to the linux ALSA driver to be routed to other applications using AudioJack or even to PulseAudio:
nc -u -p 8266 --l | play -t raw -r 44100 -b 16 -c 2 -e signed-integer -

The ESP8266 receiver as I wrote, receives the audio stream and send to external audio DAC, then the audio will go to headphones or RCA LR connected to the amplifer, as DAC I've used the Raspberry Pimoroni PHAT DAC, the ESP8266 i2S API permits max 16 bits audio (so no 24 bits), I will try next days on ESP32 that have a better i2S API and support 2 I2s ports with configurable pins and a 24 bits audio up 192Khz, I think this not a problem because I wrote an audio library for ESP8266 and ESP32, for the first tested and works well, for the last I just need to test by connecting the DAC.

My code can stream up 192Khz 16 bit Stereo audio, but the max quality I've tried successfully is 96Khz 16 bits Stereo that is high quality audio too, I think this is a limitation of ESP8266 that read the audio from microsd and not anyway referred to the network bandwith.

In my implementation one device work as trasmitter and other as receiver, not fullduplex, an interesting solution is to use both on both side, like a walkie-talkie or network audio call, on this direction 'I wrote a fast hardware optimized library for SSD1331 (96x64) RGB color small 0.95 inch oled for ESP8266 and ESP32 capable to play videos with very high framerate, up 140 FPS without audio or up 24 FPS with 44100 16 bits audio (with DAC attached) and using ESP32-CAM, i2S DAC, i2S mic etc... I want to implement an UDP video call (Audio + Video) over network between two ESPs or ESP-Desktop devices.

I noticed you warn up users for some losed packets using UDP protocol, from my tests the problem was using a broadcast address to stream to all devices, using an unicastast address I noticed that (at least on my network using FritzBox router) there is a very small amount of missed packets, like 1 missed packet every 30.000 packets, not bad.

I do not use B4R, it is installed but I use very old PCs to develop, with Win XP ?? and I've problems with it, any time I compile it recompile all the code without use a temporary file, so require very long time like the first time I compile on Arduino IDE, after this my systems are 32 bits so I cannot install anyway latest releases of the IDE. Maybe when I release my C++ code for ESP8266 and ESP32 someone interested to port it to B4R.

Please try my suggestion with VLC and Netcat if you use Linux and have some time, even on Raspberry will work, and inform me if you have success with it.

Regards
Massimo from Italy

I'm actually trying to make an walkie talkie .
Where 1 side is esp32 with mic and speaker
Other side is application.

The esp32 device is working perfectly fine.
1. It sends audio data with mic (inmp441) via UdP
2. Recive data and plays on speaker .

What i want now.
1. Application on phone which receive data and plays.
2. It should be able to record voice from mobile and send it to device so that it plays on speaker


Ultimately 1side is device other side is app

So i need your help with application which is build 70% but causing errors

 

[AISH@123]

Hi @Star-Dust , I've developed a similar project some time ago, but I use ESP8266 and ESP32 that read audio wave from microsd card and stream it over UDP to Android (B4A) or desktop devices, PC and Raspberry Pi (B4J) or another ESP8266 with external i2S audio DAC. Tried it not only in local network but even over internet from one city to other city and works pretty well. I will release the full (Arduino IDE, B4A, B4J) projects next days.

I want suggest that you can receive the UDP stream even with VLC or Netcat, with last on Linux you can pass directly the audio stream to the linux ALSA driver and then use AudioJack to route it to other applications and even use audio plugins to process it in realtime, like equalizer, reverb, delay and so... or just to record it eg. with Audacity, Ardour etc...

On linux using VLC to receive the audio stream I use the follow command on terminal that start an UDP server that listen to:
vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=44100 udp://@:8266
I've used port 8266, you can chànge it and other parameters as you need based on the trasmitter

Using Netcat the follow command pass the received audio stream to the linux ALSA driver to be routed to other applications using AudioJack or even to PulseAudio:
nc -u -p 8266 --l | play -t raw -r 44100 -b 16 -c 2 -e signed-integer -

The ESP8266 receiver as I wrote, receives the audio stream and send to external audio DAC, then the audio will go to headphones or RCA LR connected to the amplifer, as DAC I've used the Raspberry Pimoroni PHAT DAC, the ESP8266 i2S API permits max 16 bits audio (so no 24 bits), I will try next days on ESP32 that have a better i2S API and support 2 I2s ports with configurable pins and a 24 bits audio up 192Khz, I think this not a problem because I wrote an audio library for ESP8266 and ESP32, for the first tested and works well, for the last I just need to test by connecting the DAC.

My code can stream up 192Khz 16 bit Stereo audio, but the max quality I've tried successfully is 96Khz 16 bits Stereo that is high quality audio too, I think this is a limitation of ESP8266 that read the audio from microsd and not anyway referred to the network bandwith.

In my implementation one device work as trasmitter and other as receiver, not fullduplex, an interesting solution is to use both on both side, like a walkie-talkie or network audio call, on this direction 'I wrote a fast hardware optimized library for SSD1331 (96x64) RGB color small 0.95 inch oled for ESP8266 and ESP32 capable to play videos with very high framerate, up 140 FPS without audio or up 24 FPS with 44100 16 bits audio (with DAC attached) and using ESP32-CAM, i2S DAC, i2S mic etc... I want to implement an UDP video call (Audio + Video) over network between two ESPs or ESP-Desktop devices.

I noticed you warn up users for some losed packets using UDP protocol, from my tests the problem was using a broadcast address to stream to all devices, using an unicastast address I noticed that (at least on my network using FritzBox router) there is a very small amount of missed packets, like 1 missed packet every 30.000 packets, not bad.

I do not use B4R, it is installed but I use very old PCs to develop, with Win XP ?? and I've problems with it, any time I compile it recompile all the code without use a temporary file, so require very long time like the first time I compile on Arduino IDE, after this my systems are 32 bits so I cannot install anyway latest releases of the IDE. Maybe when I release my C++ code for ESP8266 and ESP32 someone interested to port it to B4R.

Please try my suggestion with VLC and Netcat if you use Linux and have some time, even on Raspberry will work, and inform me if you have success with it.

Regards
Massimo from Italy

Hi @max123 i have been waiting for your response on the pervious reply
please check out!
pigirl.02@gmail.com

 

[AISH@123]

Hi @Star-Dust , I've developed a similar project some time ago, but I use ESP8266 and ESP32 that read audio wave from microsd card and stream it over UDP to Android (B4A) or desktop devices, PC and Raspberry Pi (B4J) or another ESP8266 with external i2S audio DAC. Tried it not only in local network but even over internet from one city to other city and works pretty well. I will release the full (Arduino IDE, B4A, B4J) projects next days.

I want suggest that you can receive the UDP stream even with VLC or Netcat, with last on Linux you can pass directly the audio stream to the linux ALSA driver and then use AudioJack to route it to other applications and even use audio plugins to process it in realtime, like equalizer, reverb, delay and so... or just to record it eg. with Audacity, Ardour etc...

On linux using VLC to receive the audio stream I use the follow command on terminal that start an UDP server that listen to:
vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=44100 udp://@:8266
I've used port 8266, you can chànge it and other parameters as you need based on the trasmitter

Using Netcat the follow command pass the received audio stream to the linux ALSA driver to be routed to other applications using AudioJack or even to PulseAudio:
nc -u -p 8266 --l | play -t raw -r 44100 -b 16 -c 2 -e signed-integer -

The ESP8266 receiver as I wrote, receives the audio stream and send to external audio DAC, then the audio will go to headphones or RCA LR connected to the amplifer, as DAC I've used the Raspberry Pimoroni PHAT DAC, the ESP8266 i2S API permits max 16 bits audio (so no 24 bits), I will try next days on ESP32 that have a better i2S API and support 2 I2s ports with configurable pins and a 24 bits audio up 192Khz, I think this not a problem because I wrote an audio library for ESP8266 and ESP32, for the first tested and works well, for the last I just need to test by connecting the DAC.

My code can stream up 192Khz 16 bit Stereo audio, but the max quality I've tried successfully is 96Khz 16 bits Stereo that is high quality audio too, I think this is a limitation of ESP8266 that read the audio from microsd and not anyway referred to the network bandwith.

In my implementation one device work as trasmitter and other as receiver, not fullduplex, an interesting solution is to use both on both side, like a walkie-talkie or network audio call, on this direction 'I wrote a fast hardware optimized library for SSD1331 (96x64) RGB color small 0.95 inch oled for ESP8266 and ESP32 capable to play videos with very high framerate, up 140 FPS without audio or up 24 FPS with 44100 16 bits audio (with DAC attached) and using ESP32-CAM, i2S DAC, i2S mic etc... I want to implement an UDP video call (Audio + Video) over network between two ESPs or ESP-Desktop devices.

I noticed you warn up users for some losed packets using UDP protocol, from my tests the problem was using a broadcast address to stream to all devices, using an unicastast address I noticed that (at least on my network using FritzBox router) there is a very small amount of missed packets, like 1 missed packet every 30.000 packets, not bad.

I do not use B4R, it is installed but I use very old PCs to develop, with Win XP ?? and I've problems with it, any time I compile it recompile all the code without use a temporary file, so require very long time like the first time I compile on Arduino IDE, after this my systems are 32 bits so I cannot install anyway latest releases of the IDE. Maybe when I release my C++ code for ESP8266 and ESP32 someone interested to port it to B4R.

Please try my suggestion with VLC and Netcat if you use Linux and have some time, even on Raspberry will work, and inform me if you have success with it.

Regards
Massimo from Italy

check out

 

[hatzisn]

Note: The B4J version requires the Audiolorecord library. The B4A requires the Audio library. The B4a version works better if you have headphones (avoid the return of audio) or turn off listening when you speak and vice versa ...

Where can I find the required Audiolorecord library? I have searched the forum and I cannot find it.

 

[max123]

Use it or follow jAudioRecord2 link.
https://www.b4x.com/android/forum/threads/jaudiorecord.37930/?amp=1