Developing a Data Collection Platform to work offline with GIS Maps

Hi Fam

Im working on a data collection platform, more like google forms / microsoft forms / survey app. This however needs to work with gis and work offline. This means I need to have some kind of baseline map downloaded and available to use on the app.

I kindly request some good advise on this to anyone who has done it before. The implementation of offline base maps.

My app is a web app, the admin part (project > forms > questions designs) etc.

Data Collection - can work on mobile device via URL link.

Whilst I understand that GPS can work offline, the crux is having the base map accessible offline, being able to add markers, polylines, polygons etc.

If you have done something like this, kindly DM me, even if you have ideas on how it can be achieved.

Kind Regards

Thanks in advance.

Mashy.

Hey Mashy. I think I have something I wrote in B4A years ago. It also had GPS tracking from memory. I'll dig it up.

Here you go - 2012 so been a while since I looked at it!

From memory, it downloads the image tiles for a given extent.

Also, Mashy, instead of reinventing the wheel, please look into XLSForm - https://xlsform.org/en/

This is the industry standard for defining data collection forms (eg ArcGIS Survey 123 uses it). If possible you should use this as the way to define forms for your app.

tchart said:

This is the industry standard for defining data collection forms (eg ArcGIS Survey 123 uses it). If possible you should use this as the way to define forms for your app.

Click to expand...

So for example, you design your form in excel (Google sheets), transfer that form onto let's say an android device (Google sheets is via the www), you can then use an app like ODK Collect etc to act as the middle man that takes the data you are entering onto your android device (phone, tablet, android power barcode scanner etc) and that data automatically ends up being inserted in the excel / Google sheets, where you can then do anything you like with the information?

I've come across that scenario before, but with a well known barcode scanner manufacturer integrating with Google sheets. Sadly I never got the chance to complete the work as the client needed it up and running withing a couple hours, so said no chance.

That's the closest I've come to today scenario, I need to look more into it though.

Can someone explain to me in plain English more about the advantages etc of this solution?

I hope I'm not hijacking this thread with this question...

Peter Simpson said:

Can someone explain to me in plain English more about the advantages etc of this solution?

I hope I'm not hijacking this thread with this question...

Click to expand...

Not at all. To clarify, I think Mashiane is building something like ODK Collect.

I don't know ODK Collect, but I see they have a converter here from XLSForm to ODK https://getodk.org/xlsform/

As well as tutorials, so I guess they have some support? (I watched the tutorial, you can upload XLS Forms)

https://docs.getodk.org/tutorial-first-form

XLS Form isnt an app or solution, its a "standard" to define data collection forms. You use Excel to define what data (fields) you want to collect, how they behave (e.g., numeric, text, drop-down list) and any rules (eg if question 1 is "No" then skip question 2).

An example of an XLS Form



Its used by many data collection apps, including ODK Collect (according to the docs).

You would still need to

#1 render the form (the fields etc)
#2 Implement the logic dictated by the form definition
#2 have somewhere to save the data collected

I was suggesting to Mashiane to not invent a new form definition/format but rather to leverage an industry standard.

There are benefits such as existing GUIs to create forms outside of Excel (see below) that generate an XLS Form. Below is Survey 123 Connect which lets you drag and drop to create an XLS Form

Some AI info

Form Creation & Editing Tools​

These let you build or edit XLSForm spreadsheets (often in Excel/Sheets).

• Spreadsheet software – Microsoft Excel, Google Sheets, LibreOffice Calc (for manual XLSForm editing).
• XLSForm Builder – a web tool to create/edit XLSForms.

Tools for Conversion & Validation​

Before use in data-collection apps, XLSForms often need conversion to XForms:

• XLSForm Online – official ODK tool to convert/preview XLSForms.
• XLSForm Offline – standalone converter/validator for desktop (Windows/Mac) that creates ODK XForms compatible with many platforms.
• pyxform / xls2xform – underlying library used by many tools to convert .xlsx to .xml (XForm).

Data Collection & Management Platforms​

These systems import, use, or host XLSForm surveys:

• KoboToolbox – upload XLSForms for web/mobile data collection; download/export forms.
• Open Data Kit (ODK) ecosystem – includes ODK Collect, ODK Central (server) and the infrastructure that runs XLSForms.
• SurveyCTO – XLSForm-based commercial data collection platform.
• CommCare (Dimagi) – supports XLSForm upload and data collection.
• Survey123 for ArcGIS – XLSForm support often via import/translation.
• Ona – analytics and data platform that supports XLSForm surveys.
• Secure Data Kit (SDK) – supports XLSForm usage.
• Community Health Toolkit (CHT) – uses XLSForms for health surveys/workflows.
• ActivityInfo – can import XLSForms into its form/database system (beta).
• CyberTracker – supports XLSForm as an alternative mobile data collector linked to ODK-style backends.

Mobile Apps​

Mobile data collection apps (often linked to platforms above) that run forms created from XLSForm:

• ODK Collect – Android app for offline/online collection.
• KoboCollect – Android app tied to KoboToolbox (ODK-compatible).
• CommCare app – mobile data collection.
• Third-party mobile helpers (e.g., GIC XLSForm Creation/Recovery for iOS – forms compatible with ODK).

Hello Mr @tchart

tchart said:

Not at all. To clarify...

Click to expand...

and again

tchart said:

Some AI info...

Click to expand...

I've through everything multiple times, cheers for the information.


Enjoy...

Thanks for the heads up. The current methodologies being applied is via xls forms and arcgis survey123.

Thing is, 'they' don't want that anymore.

Hello ,
I know that you are looking for a webapp solution, this was just for learning purposes on my behalf as a desktop solution.

The attached B4X example works with both B4J and also B4A. Look in the Files Manager tab and you will see 2 files, xlsx and xml. This example can use both file types, just change LoadFormFromXlsx to LoadFormFromXml and vice versa.

There's next to nothing regarding this topic on the forum, except from 2020 when you @Mashiane mentioned it.

@tchart, thank you for the heads up regarding XLSForm and its uses. I originally wrapped the xml version, then realised my mistake, I then did some research , got the xlsx version working, then I managed to get both xml and xlsx versions working in the same library by simply adding more dependencies and playing around with the converter method.

Talking about more dependencies, CLICK HERE - to download all the dependencies necessary for this XLSForm (XLSX actually) / XMLForm thingy to work correctly, then place all the dependency files into the additional libraries folders.

It might be called XLSForm, but as I previously mentioned, it reads both XLSX and XML file formats and treats them the exact same way.

Libraries:


B4J screenshot:


B4A screenshot:


SS_XLSForm

Author: Peter Simpson
Version: 1.0

• XLSForm
  • Events:
    • ConstraintError (Message As String)
    • FormFinished
    • QuestionChanged (QName As String, QLabel As String, QType As String, IsEnd As Boolean)
  • Functions:
    • AnswerCurrentQuestion (Value As String)
      Validates and saves the answer for the current question, then moves forward.
    • GetAllAnswers As Map
      Returns all top-level answers as a Map (Name -> Value).
    • GetAllQuestionNames As List
      Returns a List containing the internal names of all questions.
    • GetLanguages As List
      Returns a List of available languages.
    • GetOptions As Map
      Returns a Map of options for select questions (Label -> Value).
    • GetValue (questionName As String) As String
      Returns the answer value for a specific question name.
    • Initialize (EventName As String)
      Initialises the XLSForm engine.
    • LoadFormFromXlsx (in As java.io.InputStream)
      Loads a standard XLSForm Excel file (.xlsx) and converts it for use.
    • LoadFormFromXml (in As java.io.InputStream)
      Loads and parses an XForm from an InputStream.
    • Next
      Moves the form navigation to the next available question.
    • Previous
      Moves the form navigation to the previous available question.
    • SetLanguage (locale As String)
      Sets the form language.
  • Properties:
    • CurrentAppearance As String [read only]
      Returns the appearance attribute for the current question.
    • CurrentHint As String [read only]
      Returns the help text (hint) for the current question.
    • CurrentLabel As String [read only]
      Returns the long-text label of the current question.
    • CurrentName As String [read only]
      Returns the internal name (ID) of the current question.
    • CurrentType As String [read only]
      Returns the control type of the current question (input, select1, select, trigger, or other).
    • HasCurrentQuestion As Boolean [read only]
      Returns True if the engine is currently positioned at a valid question.
    • IsEnd As Boolean [read only]
      Returns True if the navigation has reached the end of the form.
    • IsRequired As Boolean [read only]
      Returns True if the current question is mandatory.

Enjoy...

Wow guys, this is awesome, will learn a lot from this. Thank you so much.

Yes my first current attempt at this is web based, but with the B4XDaisyUIKit, I am tempted to make it a native app, i.e. data collection part, so Im planning and executing.

So, one is able to..

1. Create one or more projects (a project can have multiple forms)
2. Create one or more forms (a form can have 1 or more questions),
3. Create questions for each form,
4. Publish the form (field workers get a URL to collect data) and
5. Data collection and export (geojson, arrgis via rest api, excek etc) depending on needs.

Perhaps lets me show you what the "forms designer" looks like. We have different question types, labels etc and during collection, the submitted data is saved to an inbox of the person and later submitted when data collection is complete It's also an attempt to show what the form will look like on both tablet and phone.

So I have been asked if they can import existing "xls forms" into this?

The plan is to make this an MVP but with offline functionality and then this "offline map functionality" we are talking about here.



I really appreciate your inputs into this.

Mashiane said:

Thanks for the heads up. The current methodologies being applied is via xls forms and arcgis survey123.

Thing is, 'they' don't want that anymore.

Click to expand...

I can sympathise, the cost keeps going up. Even ODK collect is relatively expensive for multiple users.

Mashy, a while back Esri had this thing called geoform - its now deprecated - but it built a web form based on an ArcGIS Rest service. I really liked this option but the map didnt work offline.