Hello again, world.xy

Author: jeromechoo

.gitignore

@@ -1,4 +1,8 @@
-.idea
-.DS_Store
-*.pyc
-config.py
+# Python
+.venv
+__pycache__
+.pytest_cache
+.env
+
+# Claude
+.claude/settings.local.json

AGENTS.md

@@ -0,0 +1,5 @@
+# Agent Guidelines
+
+## README Examples
+
+Whenever a code example in `README.md` is added or updated, the corresponding test must be added or updated in `tests/test_readme_examples.py`. Run `python -m pytest tests/test_readme_examples.py` to validate before considering the work complete.

CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

INSTALL.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Diffbot
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

LICENSE

@@ -1,128 +1,195 @@
-# Python Diffbot API Client
+# Diffbot Python Library
 
+Python client library for [Diffbot](https://www.diffbot.com) APIs.
 
-## Preface
-Identify and extract the important parts of any web page in Python!  This client currently supports calls to Diffbot's Automatic APIs and Crawlbot.
 
+## Installation
 
-Installation
-To install activate a new virtual environment and run the following command:
+```bash
+pip install git+https://github.com/diffbot/diffbot-python.git
+```
 
-    $ pip install -r requirements.txt
+Or, for local development:
 
-## Configuration
+```bash
+pip install -e ".[dev]"
+```
 
-To run the example, you must first configure a working API token in config.py:
+## Usage
 
-    $ cp config.py.example config.py; vim config.py;
+### Authentication
+Set your Diffbot API token in your environment or .env.
 
-Then replace the string "SOME_TOKEN" with your API token.  Finally, to run the example:
+```bash
+export DIFFBOT_API_TOKEN=<TOKEN>
+```
 
-    $ python example.py
+### Extract structured content
+```python
+from diffbot import Diffbot
 
-## Usage
+db = Diffbot(token="YOUR_TOKEN")
+data = db.extract("https://www.example.com")
+```
 
-### Article API
-An example call to the Article API:
+### Ask Diffbot LLM
+```python
+from diffbot import Diffbot
 
-```
-diffbot = DiffbotClient()
-token = "SOME_TOKEN"
-version = 2
-url = "http://shichuan.github.io/javascript-patterns/"
-api = "article"
-response = diffbot.request(url, token, api, version=2)
+db = Diffbot(token="YOUR_TOKEN")
+for chunk in db.ask([{"role": "user", "content": "What's the capital of France?"}]):
+    print(chunk, end="")
 ```
 
-### Product API
-An example call to the Product API:
+### Crawl a site for structured content
+```python
+from diffbot import Diffbot
 
-```
-diffbot = DiffbotClient()
-token = "SOME_TOKEN"
-version = 2
-url = "http://www.overstock.com/Home-Garden/iRobot-650-Roomba-Vacuuming-Robot/7886009/product.html"
-api = "product"
-response = diffbot.request(url, token, api, version=version)
+db = Diffbot(token="YOUR_TOKEN")
+for event in db.crawl("https://www.example.com", hops=1):
+    print(event)
 ```
 
-### Image API
-An example call to the Image API:
+### Query the Knowledge Graph
+```python
+from diffbot import Diffbot
 
-```
-diffbot = DiffbotClient()
-token = "SOME_TOKEN"
-version = 2
-url = "http://www.google.com/"
-api = "image"
-response = diffbot.request(url, token, api, version=version)
+db = Diffbot(token="YOUR_TOKEN")
+results = db.dql('type:Organization name:"Diffbot"')
 ```
 
-### Analyze API
-An example call to the Analyze API:
+### Web Search
+```python
+from diffbot import Diffbot
 
-```
-diffbot = DiffbotClient()
-token = "SOME_TOKEN"
-version = 2
-url = "http://www.twitter.com/"
-api = "analyze"
-response = diffbot.request(url, token, api, version=version)
+db = Diffbot(token="YOUR_TOKEN")
+results = db.web_search("diffbot knowledge graph")
+for r in results["search_results"]:
+    print(r["score"], r["title"], r["pageUrl"])
+    print(r["content"])
 ```
 
-### Crawlbot API
-To start a new crawl, specify a crawl name, seed URLs, and the API via which URLs should be processed. An example call to the Crawlbot API:
+### Entities (NLP)
+```python
+from diffbot import Diffbot
 
-```
-token = "SOME_TOKEN"
-name = "sampleCrawlName"
-seeds = "http://www.twitter.com/"
-api = "analyze"
-sampleCrawl = DiffbotCrawl(token,name,seeds=seeds,api=api)
+db = Diffbot(token="YOUR_TOKEN")
+result = db.entities("Apple CEO Tim Cook announced record quarterly earnings.")
+for entity in result["entities"]:
+    print(entity["name"], entity.get("type"), entity.get("id"))
+print("sentiment:", result.get("sentiment"))
 ```
 
-Omit "seeds" and "api" to load an existing crawl, or create a crawl as a placeholder.
+## Async Usage
 
-To check the status of a crawl:
+### Extract structured content
+```python
+import asyncio
+from diffbot import DiffbotAsync
 
-```
-sampleCrawl.status()
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        data = await db.extract("https://www.example.com")
+        print(data)
+
+asyncio.run(main())
 ```
 
-To update a crawl:
+### Ask Diffbot LLM
+```python
+import asyncio
+from diffbot import DiffbotAsync
 
-```
-maxToCrawl = 100
-upp = "diffbot"
-sampleCrawl.update(maxToCrawl=maxToCrawl,urlProcessPattern=upp)
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        async for chunk in db.ask([{"role": "user", "content": "What's the capital of France?"}]):
+            print(chunk, end="")
+
+asyncio.run(main())
 ```
 
-To delete or restart a crawl:
+### Crawl a site for structured content
+```python
+import asyncio
+from diffbot import DiffbotAsync
 
-```
-sampleCrawl.delete()
-sampleCrawl.restart()
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        async for event in db.crawl("https://www.example.com", hops=1):
+            print(event)
+
+asyncio.run(main())
 ```
 
-To download crawl data:
+### Query the Knowledge Graph
+```python
+import asyncio
+from diffbot import DiffbotAsync
 
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        results = await db.dql('type:Organization name:"Diffbot"')
+        print(results)
+
+asyncio.run(main())
 ```
-sampleCrawl.download() # returns JSON by default
-sampleCrawl.download(data_format="csv")
-```
 
-To pass additional arguments to a crawl:
+### Web Search
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        results = await db.web_search("diffbot knowledge graph")
+        for r in results["search_results"]:
+            print(r["score"], r["title"], r["pageUrl"])
+            print(r["content"])
 
+asyncio.run(main())
 ```
-sampleCrawl = DiffbotCrawl(token,name,seeds,apiUrl,maxToCrawl=100,maxToProcess=50,notifyEmail="support@diffbot.com")
+
+### Entities (NLP)
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        result = await db.entities("Apple CEO Tim Cook announced record quarterly earnings.")
+        for entity in result["entities"]:
+            print(entity["name"], entity.get("type"), entity.get("id"))
+        print("sentiment:", result.get("sentiment"))
+
+asyncio.run(main())
 ```
 
-## Testing
+## CLI
 
-First install the test requirements with the following command:
+This library also includes a CLI.
 
-    $ pip install -r test_requirements.txt
+```bash
+export DIFFBOT_API_TOKEN=your-token-here
+
+db extract https://www.example.com
+db ask "What's the capital of France?"
+db crawl https://www.example.com --hops 1
+db crawl-list-jobs
+db crawl-delete-job crawl-1234567890
+db web-search "diffbot knowledge graph"
+db web-search "diffbot knowledge graph" -n 5 -f json
+db entities "Apple CEO Tim Cook announced record quarterly earnings."
+db entities "Apple CEO Tim Cook announced record quarterly earnings." -f dql
+```
 
-Currently there are some simple unit tests that mock the API calls and return data from fixtures in the filesystem.  From the project directory, simply run:
+## Tests
 
-    $ nosetests
+Run the mock test suite:
+```bash
+python -m pytest
+```
+
+Run live integration tests against the real API (requires a valid token):
+```bash
+DIFFBOT_TOKEN=your_token python -m pytest -m live
+```