The Twitter parser is broken

[marvinborner]

The feed on alpaca.ml shows the error message Invalid date. I haven't investigated it yet but it seems like our parser is broken.

Edit: The backend behind alpaca.ml does not log anything weird and the request returns []. So it really seems like Twitter changed something.

[LarsVomMars]

https://github.com/desmondmorris/node-twitter

[marvinborner]

We would need a token for this and the whole purpose of a self-written scraper was that we can parse
without a token :)

[LarsVomMars]

Might end up being better tho :)

[marvinborner]

We could also try to fix the issue instead of throwing some nodejs libs into the project

[LarsVomMars]

Select all article tags from the page

[marvinborner]

While this seems to select the correct item, the content div doesn't exist anymore and is split up into several tags. We may have to loop over them.

[LarsVomMars]

Yeah, ik

[Mondanzo]

Leave me out of your parser mess. I make front-end not back-end D:

[marvinborner]

It seems like the parser isn't even the problem. The JSDOM call doesn't return the correct DOM as all the tweets are missing (console.log(dom.window.document.body.innerHTML).
Or I'm just dumb.

[Kurty00]

It is not recommended to scrape from Twitter since it states in the TOS that "scraping the Services without the prior consent of Twitter is expressly prohibited" (§4 Twitter TOS)

[marvinborner]

Yeah, we know. This was a temporary solution which seemed quite acceptable for a hackathon application. We could switch to the official API if someone wants to invest the time.