Capture web data

Sort by date Sort by votes

Stulish

Active Member
Licensed User
Longtime User
Gary,

I have not played with HTTP request before so threw then below example together to show you how i would do it, it is contained in the example ZIP file with the layouts. could probably do this neatening up a little, but i thought i would get it posted.

I hope it is of some use and is what you were looking for,

Regards

Stu

B4X: 
Sub Process_Globals
    Dim hc As HttpClient
End Sub
Sub Globals
   Dim StockSpin As Spinner
   Dim IndexSpin As Spinner
   Dim GetInfo As Button
   Dim Info1 As Label
   Dim Info2 As Label
   Dim Info3 As Label
   Dim Info4 As Label
   Dim Info5 As Label
   Dim Info6 As Label
   Dim Info7 As Label
   Dim Info8 As Label
   Dim Info9 As Label
   Dim Info10 As Label
   Dim Info11 As Label
   Dim req As HttpRequest
   Dim HTTPString As String 
   Dim HTTPStock As String 
   Dim HTTPComp As String
   Dim Stocks As Map 
End Sub
Sub Activity_Create(FirstTime As Boolean)
    If FirstTime Then
   Activity.LoadLayout("main")
   Activity.Title="Stock Check Example"
        hc.Initialize("hc")
      Stocks.Initialize
      HTTPString = "http://finance.google.com/finance/info?client=ig&q="
         HTTPStock = "NASDAQ:"
      HTTPComp = "FLWS" 'NASDAQ Composite
      Stocks.Put("FLWS","1-800-FLOWERS.COM, Inc.")
      Stocks.Put("FCTY","1st Century Bancshares, Inc.")
      Stocks.Put("FCCY","1st Constitution Bancorp")
      Stocks.Put("SRCE","1st Source Corporation")
      Stocks.Put("FUBC","1st United Bancorp Inc (Florida)")
      Stocks.Put("VNET","21Vianet Group Inc")
      Stocks.Put("SSRX","3SBio Inc. (ADR)")
      Stocks.Put("JOBS","51job, Inc. (ADR)  ")
      Stocks.Put("EGHT","8x8, Inc.")
      For a=0 To Stocks.Size-1
      StockSpin.Add(Stocks.GetValueAt(a))
      Next
      IndexSpin.Add("NASDAQ")
   End If
End Sub
Sub hc_ResponseSuccess (Response As HttpResponse, TaskId As Int)
    Dim rS As String, list1 As Map 
   Dim strt As Int 
   Dim started As Boolean 
   Dim dataList As List 
   dataList.Initialize 
    rS = Response.GetString("UTF8")
    'Work with the result
   Log(Response.StatusCode)
   Log ("Length:" & rS.Length)
   Log(rS)
   strt = 0: started = False
   For a=0 To rS.Length-1 
      If rS.CharAt(a) = Chr(34) AND started = False Then
         strt = a+1
         started = True
      Else
         If rS.CharAt(a) = Chr(34) AND started = True Then
            started = False
            dataList.Add(rS.SubString2(strt,a))
         End If
      End If
   Next
   Log (dataList)   
   Display(dataList)
End Sub
Sub hc_ResponseError (Response As HttpResponse, Reason As String, StatusCode As Int, TaskId As Int)
    Log("Error connecting: " & Reason & " " & StatusCode)
    If Response <> Null Then
        Log(Response.GetString("UTF8"))
        Response.Release
    End If
End Sub 
Sub StockSpin_ItemClick (Position As Int, Value As Object)
   HTTPComp = Stocks.GetKeyAt(Position)
End Sub
Sub GetInfo_Click
   req.InitializeGet(HTTPString & HTTPStock & HTTPComp)
      hc.Execute(req, 1)
End Sub
Sub Display(Info As List)
Info1.Text = Info.Get(1)
Info2.Text = Info.Get(3)
Info3.Text = Info.Get(5)
Info4.Text = Info.Get(7)
Info5.Text = Info.Get(9)
Info6.Text = Info.Get(11)
Info7.Text = Info.Get(13)
Info8.Text = Info.Get(15)
Info9.Text = Info.Get(17)
Info10.Text = Info.Get(19)
Info11.Text = Info.Get(21)
End Sub
Sub IndexSpin_ItemClick (Position As Int, Value As Object)
   HTTPStock = Value & ":"
End Sub
 

Attachments

  • stock check HTTP example.zip
    61.2 KB · Views: 320
Last edited:
Upvote 0
G

GaryK4

Member
Licensed User
Longtime User
Stu,

I think this will be very helpful.
There is discussions that Google finance may stop as well, but I wiil worry about it when that happens.

Thanks for the fast and helpful response!

Gary...
 
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
I updated the code using Yahoo Finance and Yahoo data download site. This site explains it all. Attached is also a copy of the updated code and layout file.

i have used the same string as from the website above:

for example: http://finance.yahoo.com/d/quotes.csv?s=XOM+BBDb.TO+JNJ+MSFT&f=snd1l1yr

items above in bold and italic have been removed so you can select the company from the Spinner

the letters in other colours are defined in the quote below, just ass the letters for the extra information.

B4X: 
Sub Process_Globals
    Dim hc As HttpClient
End Sub
Sub Globals
   Dim StockSpin As Spinner
   Dim GetInfo As Button
   Dim Info1 As Label
   Dim Info2 As Label
   Dim Info3 As Label
   Dim Info4 As Label
   Dim Info5 As Label
   Dim Info6 As Label
   Dim req As HttpRequest
   Dim HTTPString As String 
   Dim HTTPStock As String 
   Dim HTTPComp As String
   Dim Stocks As Map 
End Sub
Sub Activity_Create(FirstTime As Boolean)
    If FirstTime Then
   Activity.LoadLayout("main")
   Activity.Title="Stock Check Example"
        hc.Initialize("hc")
      Stocks.Initialize
      HTTPString = "http://finance.yahoo.com/d/quotes.csv?s="
      HTTPComp = "FLWS" 
      Stocks.Put("FLWS","1-800-FLOWERS.COM, Inc.")
      Stocks.Put("FCTY","1st Century Bancshares, Inc.")
      Stocks.Put("FCCY","1st Constitution Bancorp")
      Stocks.Put("SRCE","1st Source Corporation")
      Stocks.Put("FUBC","1st United Bancorp Inc (Florida)")
      Stocks.Put("VNET","21Vianet Group Inc")
      Stocks.Put("SSRX","3SBio Inc. (ADR)")
      Stocks.Put("JOBS","51job, Inc. (ADR)  ")
      Stocks.Put("EGHT","8x8, Inc.")
      For a=0 To Stocks.Size-1
      StockSpin.Add(Stocks.GetValueAt(a))
      Next
   End If
End Sub
Sub hc_ResponseSuccess (Response As HttpResponse, TaskId As Int)
    Dim rS,st As String
   Dim strt As Int 
   Dim dataList As List 
   dataList.Initialize 
    rS = Response.GetString("UTF8")
    'Work with the result
   strt = 0
   For a=0 To rS.Length-1 
      If rS.CharAt(a) = "," Then
         st = rS.SubString2(strt,a)
         st = st.Replace(Chr(34), "")
         dataList.Add(st.Trim)
         strt = a+1
      End If
   Next
   st = rS.SubString(strt)
   st = st.Replace(Chr(34), "")
   dataList.Add(st.Trim)
   Display(dataList)
End Sub
Sub hc_ResponseError (Response As HttpResponse, Reason As String, StatusCode As Int, TaskId As Int)
    Log("Error connecting: " & Reason & " " & StatusCode)
    If Response <> Null Then
        Log(Response.GetString("UTF8"))
        Response.Release
    End If
End Sub 
Sub StockSpin_ItemClick (Position As Int, Value As Object)
   HTTPComp = Stocks.GetKeyAt(Position)
End Sub
Sub GetInfo_Click
   req.InitializeGet(HTTPString & HTTPComp & "&f=snd1l1yr") ' see table for what data you want to extract
      hc.Execute(req, 1)
End Sub
Sub Display(Info As List)
'See Table for Symbols
Info1.Text = Info.Get(0) ' s = Symbol
Info2.Text = Info.Get(1) ' n = Name
Info3.Text = Info.Get(2) ' d1 = Last Trade Date
Info4.Text = Info.Get(3) ' l1 = Last Trade (Price Only)
Info5.Text = Info.Get(4) ' y = Dividend Yield
Info6.Text = Info.Get(5) ' r = P/E Ratio
End Sub



hopefully someone may be able to use this

Regards

stu
 

Attachments

  • stock check HTTP example update1.zip
    60.9 KB · Views: 343
Last edited:
Upvote 0
G

GaryK4

Member
Licensed User
Longtime User
Stu,

I did have yahoo working. However yahoo had some kind of contract problem with DOW. ^DJI (DOW) is no longer supported. I can get the data from Google finance unless they stop in the future as well. Yahoo works for all securities, just not DOW.

Thanks again for all you help!
Have a happy New Year!
 
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
Error Getting Returned Data

Hi guys,

I have been playing with the HTTPRequest and when i sent "http://eu.battle.net/api/wow/character/aszune/sturinde?fields=items" i get the error in the picture, the log can be seen below along with the code.


B4X: 
Sub Process_Globals
    Dim hc As HttpClient
   Dim req As HttpRequest
End Sub
Sub Globals
End Sub
Sub Activity_Create(FirstTime As Boolean)

    hc.Initialize("hc")
   req.InitializeGet("http://eu.battle.net/api/wow/character/aszune/sturinde?fields=items")
      hc.Execute(req, 1)
End Sub
Sub hc_ResponseSuccess (Response As HttpResponse, TaskId As Int)
    Dim rS As String
    rS = Response.GetString("UTF8")
   Log(rS)
End Sub
Sub hc_ResponseError (Response As HttpResponse, Reason As String, StatusCode As Int, TaskId As Int)
    Log("Error connecting: " & Reason & " " & StatusCode)
    If Response <> Null Then
        Log(Response.GetString("UTF8"))
        Response.Release
    End If
End Sub



if i type the above url into a web browser i get:

This is all the data for one of my World of Warcraft Characters, i was wondering why this causes an error and is there a way to get the data into a string or even better a map, the site does say its output is JSON (API info at found HERE).

regards

Stu
 

Attachments

  • error-network.png
    44.8 KB · Views: 348
Upvote 0

mc73

Well-Known Member
Licensed User
Longtime User
Not sure, but I think that if you move your code to a service module, this error will disappear. Seems like a security thing to me.
 
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
Guys,

I have had some joy with getting information from the blizzard server, basically if i just ask for basic char info i get a map of the data in accordance with their website information. if i then ask for all the items the character has i get an error and looking at the returned string, it isn't the same as what is on their site.

being new to the JSON data structure i was wondering what i am doing wrong.

i have attached the sample i am using it just sends the information to log so nothing is displayed on the screen.

B4X: 
'Activity module
Sub Process_Globals

End Sub

Sub Globals
Dim CharData As Map 
End Sub

Sub Activity_Create(FirstTime As Boolean)
   Dim job1 As HttpJob
   CharData.Initialize 
   job1.Initialize("Job1", Me)
   job1.download("http://eu.battle.net/api/wow/character/aszune/sturinde")            'this job works
   'job1.download("http://eu.battle.net/api/wow/character/aszune/sturinde?fields=items") 'this job doesnt
End Sub
Sub JobDone (Job As HttpJob)
Dim retInf As String
Dim s1, s2,s3,s4 As Int 
CharData.Clear 
   Log("JobName = " & Job.JobName & ", Success = " & Job.Success)
   If Job.Success = True Then
      Select Job.JobName
         Case "Job1"
         retInf = Job.GetString
         Log(retInf)
         get_data(retInf)
      End Select
   Else
      Log("Error: " & Job.ErrorMessage)
      ToastMessageShow("Error: " & Job.ErrorMessage, True)
   End If
   Job.Release
End Sub
Sub Activity_Resume

End Sub
Sub Activity_Pause (UserClosed As Boolean)

End Sub
Sub get_data(data As String)
Dim JSON As JSONParser
Dim Map1 As Map
JSON.Initialize(data)
Map1 = JSON.NextObject
Log(Map1)
End Sub

any ideas greatly appreciated.

Thanks

Stu
 

Attachments

  • wow check.zip
    21.7 KB · Views: 298
Upvote 0

lagore

Active Member
Licensed User
Longtime User
Hi I just tried that 'Get' request using the full request to get the items and it worked fine, not sure why your code is not working.
 
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
did you comment out the first get statement and un-comment the second.

B4X: 
job1.download("http://eu.battle.net/api/wow/character/aszune/sturinde")                'this job works
'job1.download("http://eu.battle.net/api/wow/character/aszune/sturinde?fields=items") 'this job doesnt

The first one works fine but the second one doesn't (the second statement has the useable data)

Thanks

Stu
 
Upvote 0

lagore

Active Member
Licensed User
Longtime User
Ya it was the second longer one with the data that I used, I did not use your code but a simple test app using the same URL and it worked fine, will down your code and try it out.

edit. just ran your code and it worked perfectly it loaded the Map with the returned data.

 
Last edited:
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
Thanks guys,

I am not sure why it didn't work before, but i just ran it on my Galaxy S3 and nexus 7 and the results were correct ???


not sure why it displayed an error before considering i haven't changed the code

Regards

Stu
 
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
Hi again guys (i think i am going mad with this), everything was working great and i managed to get each item into a different map, i have now changed the app to let the user select the realm and character, but i get the error i had earlier again on both of my android devices.

i have attached the code, and am now pulling my hair out :BangHead:

Not sure what is going on the log is below for the error:


thanks

Stu
 

Attachments

  • WOWCheck.zip
    263.6 KB · Views: 280
Upvote 0

Stulish

Active Member
Licensed User
Longtime User
you can even click on the link in the log quote above that was created with the app and it opens a webpage with the information??

its as if i am asking for the data before it has been returned from the server

any thoughts????
 
Upvote 0
You must log in or register to reply here.

Similar Threads

I
  • Article
B4A Library UltimateWebView Custom View
4 5 6
Replies
116
Views
52K
Cesaral
C
Mashy Teaches WebApp/Website Development with BANanoVuetifyAD3 - The New Series
7 8 9
Replies
163
Views
74K
Mashiane
  • Article
Android Example Signature Capturing with jSignature and WebViewExtras
Replies
2
Views
6K
josejad
G
  • Question
Android Question Capture web page text
Replies
2
Views
991
DonManfred
  • Question
WebView CaptureBitmap
Replies
8
Views
6K
bluedude
B
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more…