Skip to content

refactored to use google places search url instead of local services listing which is now not available#7

Merged
christivn merged 2 commits intochristivn:mainfrom
devsanthoshmk:main
Apr 15, 2026
Merged

refactored to use google places search url instead of local services listing which is now not available#7
christivn merged 2 commits intochristivn:mainfrom
devsanthoshmk:main

Conversation

@devsanthoshmk
Copy link
Copy Markdown
Contributor

I implemented the scraper using the discussed methods in #5. I wrote the core scrapping logic and methods on my own and used claude code implement that part in mapScraper/placesCrawlerV2.py and i read every line of the code. everything is working to my eyes and i can change if you want anything to. Sorry for the commit history i was using this in some other places so i just pasted them here instead of preserving the commit history.

Here is how i implemented it:

I get the search results from here
https://www.google.com/search?q={query}&start={pagination}&udm=1&hl={lang}&gl={country}

  • The udm param opens the places tab in google results directly.
  • This returns the data backed into HTML and i scrapped it using the reliable class names, IDs and other patterns.
  • Some results have no phone numbers in them so i used a separate call to the google async route https://www.google.com/async/lcl_akp?q={query}&async=ludocids:{cid},_fmt:prog with the cid from the previous result by which we can get phone numbers, complete address and website link of the places(if phone number is not there then website is also not present).
  • Here comes the modes fast,strandard,complete (refer readme for more info on this) which will handle the missing phone number problem.
  • I made everthing async too.

- Created a new `mapScraper` package with an `__init__.py` for better structure.
- Implemented `places_crawler.py` which now uses different url for seatching of Google Maps Places.
- Updated `mapScraperX.py` to support new scraping modes: fast, standard, and complete.
- Updated `requirements.txt` to include `phonenumbers` for phone validation.
@christivn
Copy link
Copy Markdown
Owner

Hi @devsanthoshmk , thank you so much for your contribution to the project. Great work!

I don't have much time to write code right now, but I hope to see more pull requests from you with new features or improvements for the project 💪

Thx <3

@christivn christivn merged commit 07e5c0b into christivn:main Apr 15, 2026
@christivn christivn mentioned this pull request Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants