web dat for password protected page

Hi,
I’m trying to using web dat to get data from an html page that requires authentication.
In that same page I have the user and password input forms and a button to authenticate.

Is there a way to fill the input forms and simulate the mouseclick on the button before fetching the data?
or a way to send user and password info to simulate the auto-authenticate behaviour of the browser…

this is the html of the input forms and button

Hey,

it might be possible to go directly to the authentication page.
First, the url for the web DAT should be the one that is set in the html form tag.
Next, use a Table DAT as the first input to the Web DAT and in 2 rows and columns specify the username and password with the first column being the input names (like “username” and “password”) and the second column containing the values for these input boxes.

Best
Markus

Hi Markus,
thanks for the answer. I managed to login but then I realized that the data that i want to scrape from the web needs flash player to be loaded. Using the webbrowser component from the Palette (which - I assume - supports flash), I finally get to visualize - and interact with - the data as a TOP rendered image. Is there a way to export to a DAT that html code that the WEBBROWSER from the Palette is reading?
I saw in the palette another component WEBCOMP that seems promising but I have no clue about getting it working in this kind of scenario…
anyone can point me to a possible solution?

Hey,

I actually didn’t know that the WebTOP supported flash - that’s surprising to me.
The WebDAT is the “source view” to the webTOP to some extend. It doesn’t handle redirects etc but shows the same HTML that the webTOP renders.

Do you have an example of the data source? I’d be interested in taking a look at it.
Generally though, if it is contained in a flash file, as far as i’m aware you would not have access to it via the page’s html source.

Can you see the data if you look at the page source in your OS’s browser?

Best
Markus

Hi Markus,
you are most probably right about flash support.
Probably it’s better if a take a step back and I try to describe the project I’m working on.
I need to get data from different sites, the one I’m struggling with is CAM4, the popular erotic cam website (hope none getting :open_mouth: :blush: ).
After logging in, when you connect to one of the thounsands of available cam rooms, you can visualize - above the chat - the number of users connected to that specific cam room, that is the data I’m trying to obtain which is nested in regular html:

  • Users [b]465[/b]
  • I attach a toe file with:

    • a web top, a web dat, a WEBBROWSER COMP.
    • a table dat with username and password for a CAM4 account I created
    • few screenshot of a standard cam

    As the cams are not available 24hours, the one I have linked in the attached toe could be not available if someone is trying to test it, so you should connect to cam4.com with a standard browser, click on any of the listed cam, get the correspondent link and copy in the web top and dat.

    If I feed the web dat with a table with username and password as you suggested, I think that instead of logging in and connecting to the selected chat room URL, I believe the web dat loads the home page of my account that is it.cam4.com/totalusers

    I succeded to navigate to the cam room, log in and visualize the total users, using the webbrowser comp from the palette, because it allows to interact with the website as in a regular browser but I cannot access the html under the hood or extract text from it.
    any advise?
    totaluser.toe (186 KB)

    Hm, I would expect that you can login via the webCOMP and using Basic Authentication.

    Sorry - not able to try that out though…

    Best
    Markus

    Thanks Markus,
    I tried the web com with all sort of authentication but I think I miss the way of properly using this comp, as I can’t succed in gettin any html data even inserting any no-login web address.
    anyone can point me in the right direction?

    Just saw that the Session get’s not activated by default. You have to toggle the Active Flag first to make this work.

    Will fix this for the next release.

    Best
    Markus