Introduction
Since the release of OpenAI's ChatGPT, and later its REST API, the technology world has been enveloped by the vast capabilities of this paradigm-shifting technology and pushing it to its limits. However, for as many strengths as it has, ChatGPT has two pronounced weaknesses: 1) its training data cutoff is 2021, meaning it has no knowledge of information in more recent times, and 2) it is prone to hallucinations or making up information.
To circumvent this issue, software developers and prompt engineers alike have taken on the task of supplementing ChatGPT's knowledge base with additional information that can provide more relevant, reliable, and informed responses. This process involves a combination of technologies and prompt engineering that we'll discuss in depth in this series.
In this article, we'll investigate how we can write a program to scrape the internet for content that we can use to supplement an LLM like ChatGPT's knowledge beyond its training data. Specifically, we'll gather information from Nourish by WebMD and process articles to create a databank of sources on nutrition that will be used to power a nutrition chatbot.
This is part one of my series on creating a nutrition chatbot powered by data from WebMD using sentence-transformers, FAISS, and ChatGPT.
Part 1: Web-Scraping Techniques to Source Reliable Information for ChatGPT
Part 2: Leveraging Vector Embeddings and Similarity Search to Supplement ChatGPT's Training Data (coming soon)
Part 3: ChatGPT Prompt-Engineering Techniques for Providing Contextual Information (coming soon)
Nourish by WebMD provides a plethora of articles related to nutrition and eating well. Each article is conveniently broken up into multiple sub-headings, with the text underneath each heading providing additional relevant information to the main article topic. Since each sub-heading contains specific information on a sub-topic related to the main topic, instead of choosing the save the entire article as one "document" so to say, we'll want to break the article up by headings into multiple documents.
All of the articles are listed on WebMD's Health & Diet Medical Reference, which is paginated to 23 pages at the time of writing. Each page in this reference contains additional articles that we will need to scrape.
To perform the web scraping, we'll use the BeautifulSoup library, as well as the built-in requests library. Recall that we have to iterate through each paginated page in the Reference. We'll then use html2text to convert the plain HTML of each page into markdown content and split the page by markdown headings into multiple documents.
You can do this project in a Jupyter Notebook, or on a platform like Google Colab. Alternatively, you can create a new Python project in an IDE like Pycharm or VS Code. I recommend using a Jupyter Notebook for following along with this article.
The code in this article is also available for you to follow along (and run yourself) on Google Colab.
Getting Started
Let's first install the necessary dependencies for this project:
pip install bs4 html2text
Next, we'll define some custom request headers to spoof a web-browser client that we will use when making requests:
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Accept-Language': 'en-US,en;q=0.9',
}
Crawling through the Reference
Because each article we want to scrape is located within WebMD's Health & Diet Medical Reference, and the Reference itself is paginated, we need to first create a web-crawler program to identify each link we will need to scrape.
To do so, let's first make a get_links_from_page function that will retrieve all of the links to articles from a specific page within the Reference:
import requests
from bs4 import BeautifulSoup
def get_links_from_page(index):
url = f"https://www.webmd.com/diet/medical-reference/default.htm?pg={index}"
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
ul_element = soup.find('section', class_='dynamic-index-feed').find('ul', class_='list')
anchors = ul_element.find_all('a')
links = [anchor['href'] for anchor in anchors]
return links
Next, we'll create a get_all_links_from_reference function that will iterate through all 23 pages in the Reference and fetch all of the links to articles.
all_links = []
def get_all_links_from_reference():
for index in range(1, 24):
all_links.extend(get_links_from_page(index))
Once we run that, we now have all of the links in the WebMD Health & Diet Reference, and we can now move on to scraping the individual articles for their content.
Scraping and Processing Articles
As previously discussed, we will break up each article into multiple "documents" separated by headings within the article. Let's make a utility Document class to help us structure our code and keep track of the title and URL of the article, as well as the content of the document:
class Document:
def __init__(self, title, url, content):
self.title = title
self.url = url
self.content = content
We'll also write a brief function that utilizes requests and BeautifulSoup to scrape a given article via its URL:
import requests
from bs4 import BeautifulSoup
def scrape_webpage(url):
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
return soup
We now want to begin the work on processing the HTML into documents. First, let's write a function that converts the webpage into markdown text:
from html2text import html2text
def webpage_to_markdown(soup):
article_body = soup.find(class_='article__body')
if article_body is None:
article_body = soup.find(class_='article-body')
html_content = str(article_body)
markdown_content = html2texthtml2text(html_content, bodywidth=0)
return markdown_content
For the sake of making the content more legible for ChatGPT, we'll make a function that strips links from the markdown content and replaces them with just the text of the link. We can do so using regular expressions, like so:
import re
def strip_links_from_markdown(markdown_content):
link_pattern = re.compile(r'\[(.*?)\]\((.*?)\)')
matches = link_pattern.findall(markdown_content)
for _match in matches:
full_match = f'[{_match[0]}]({_match[1]})'
markdown_content = markdown_content.replace(full_match, _match[0])
return markdown_content
Lastly, split the articles by markdown headings into multiple documents.
def split_markdown_by_headings(markdown_content):
pattern = r'(#{1,2}.*)'
sections = re.split(pattern, markdown_content)
combined_sections = []
for i in range(1, len(sections), 2):
heading = sections[i].strip()
text = sections[i + 1].strip() if i + 1 < len(
sections) else ''
combined_section = f"{heading}\n{text}"
combined_sections.append(combined_section)
if len(combined_sections) == 0:
combined_sections = [markdown_content]
return combined_sections
We now have all of the logic to both scrape and process articles from the Reference. We can put it together in a function, create_documents_from_webpage:
def create_documents_from_webpage(url):
try:
soup = scrape_webpage(url)
title = soup.find('h1').get_text().strip()
markdown_content = webpage_to_markdown(soup)
markdown_content = strip_links_from_markdown(markdown_content)
contents_by_heading = split_markdown_by_headings(markdown_content)
docs = []
for content in contents_by_heading:
docs.append(Document(title=title, url=url, content=content))
return docs
except:
return []
With this function, we can provide any URL from the Reference, and it will generate a list of documents with content from each of the subheadings.
Generate Databank of Sources
Now that we have all of the logic required to scrape and process articles, we can move on to generating the databank of nutritional sources that will be utilized by the chatbot. To do so, we must iterate through each of the articles on of the pages in the Reference, and save all of the documents created.
We'll put together all of our previous work in the generate_databank function:
def generate_databank():
docs = []
all_links = get_all_links_from_reference()
for link in all_links:
docs_from_article = create_documents_from_webpage(link)
docs.extend(docs_from_article)
return docs
Feel free to add additional debug comments, such as which link is currently being processed, how many links are left, how many documents were added, etc.
After running generate_databank, we now have a databank of nutrition sources!
Keep in mind that this may be a memory-intensive process, and depending on your machine, you may want to consider running this program in a hosted environment like Google Colab. I was able to run this program on my 2023 M2 Pro MacBook Pro, but results may vary by your specific setup.
Conclusion
Through following the steps in this article, we utilized real-world web-scraping techniques to gather and process information from the internet and create a databank of sources; this databank will be used to provide reliable information to our nutrition chatbot.
We used the BeautifulSoup library and the requests library to crawl through the WebMD Health & Diet Medical Reference, and wrote functions to convert the webpage into markdown text, strip links from the markdown content, and split the articles by markdown headings. Finally, we generated the databank of nutritional sources by iterating through all of the articles on pages in the Reference and saving all of the documents created.
In the next article in this series, we'll explore using sentence-transformers and FAISS to create a vector store index to connect the databank we created to an LLM like ChatGPT.
