Since the release of OpenAI's ChatGPT, and later its REST API, the technology world has been enveloped by the vast capabilities of this paradigm-shifting technology and pushing it to its limits. However, for as many strengths as it has, ChatGPT has two pronounced weaknesses: 1) its training data cutoff is 2021, meaning it has no knowledge of information in more recent times, and 2) it is prone to hallucinations or making up information.

To circumvent this issue, software developers and prompt engineers alike have taken on the task of supplementing ChatGPT's knowledge base with additional information that can provide more relevant, reliable, and informed responses. This process involves a combination of technologies and prompt engineering that we'll discuss in depth in this series.

In this article, we'll investigate how we can write a program to scrape the internet for content that we can use to supplement an LLM like ChatGPT's knowledge beyond its training data. Specifically, we'll gather information from Nourish by WebMD and process articles to create a databank of sources on nutrition that will be used to power a nutrition chatbot.

This is part one of my series on creating a nutrition chatbot powered by data from WebMD using sentence-transformers, FAISS, and ChatGPT.

• Part 1: Web-Scraping Techniques to Source Reliable Information for ChatGPT

• Part 2: Leveraging Vector Embeddings and Similarity Search to Supplement ChatGPT's Training Data (coming soon)

• Part 3: ChatGPT Prompt-Engineering Techniques for Providing Contextual Information (coming soon)

Nourish by WebMD provides a plethora of articles related to nutrition and eating well. Each article is conveniently broken up into multiple sub-headings, with the text underneath each heading providing additional relevant information to the main article topic. Since each sub-heading contains specific information on a sub-topic related to the main topic, instead of choosing the save the entire article as one "document" so to say, we'll want to break the article up by headings into multiple documents.

All of the articles are listed on WebMD's Health & Diet Medical Reference, which is paginated to 23 pages at the time of writing. Each page in this reference contains additional articles that we will need to scrape.

To perform the web scraping, we'll use the BeautifulSoup library, as well as the built-in requests library. Recall that we have to iterate through each paginated page in the Reference. We'll then use html2text to convert the plain HTML of each page into markdown content and split the page by markdown headings into multiple documents.

You can do this project in a Jupyter Notebook, or on a platform like Google Colab. Alternatively, you can create a new Python project in an IDE like Pycharm or VS Code. I recommend using a Jupyter Notebook for following along with this article.

The code in this article is also available for you to follow along (and run yourself) on Google Colab.

Getting Started

Let's first install the necessary dependencies for this project:

pip install bs4 html2text

Next, we'll define some custom request headers to spoof a web-browser client that we will use when making requests:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Accept-Language': 'en-US,en;q=0.9',
}

Crawling through the Reference

Because each article we want to scrape is located within WebMD's Health & Diet Medical Reference, and the Reference itself is paginated, we need to first create a web-crawler program to identify each link we will need to scrape.

To do so, let's first make a get_links_from_page function that will retrieve all of the links to articles from a specific page within the Reference:

import requests
from bs4 import BeautifulSoup


def get_links_from_page(index):
  # Send a GET request to the page we want to crawl
  url = f"https://www.webmd.com/diet/medical-reference/default.htm?pg={index}"
  response = requests.get(url, headers=headers)

  # Parse the HTML content using BeautifulSoup
  soup = BeautifulSoup(response.content, 'html.parser')

  # Find the ul element with the links to articles
  ul_element = soup.find('section', class_='dynamic-index-feed').find('ul', class_='list')

  # Find all the anchors (a tags) within the ul element
  anchors = ul_element.find_all('a')

  # Extract the href attribute from each link
  links = [anchor['href'] for anchor in anchors]

  return links

Next, we'll create a get_all_links_from_reference function that will iterate through all 23 pages in the Reference and fetch all of the links to articles.

all_links = []

def get_all_links_from_reference():
  for index in range(1, 24):
    all_links.extend(get_links_from_page(index))

Once we run that, we now have all of the links in the WebMD Health & Diet Reference, and we can now move on to scraping the individual articles for their content.

Scraping and Processing Articles

As previously discussed, we will break up each article into multiple "documents" separated by headings within the article. Let's make a utility Document class to help us structure our code and keep track of the title and URL of the article, as well as the content of the document:

class Document:
  def __init__(self, title, url, content):
    self.title = title
    self.url = url
    self.content = content

We'll also write a brief function that utilizes requests and BeautifulSoup to scrape a given article via its URL:

import requests
from bs4 import BeautifulSoup


def scrape_webpage(url):
  response = requests.get(url, headers=headers)
  soup = BeautifulSoup(response.content, 'html.parser')
  return soup

We now want to begin the work on processing the HTML into documents. First, let's write a function that converts the webpage into markdown text:

from html2text import html2text


def webpage_to_markdown(soup):
  # Find the article body, either by `class_='article__body'` or `class_='article-body'`
  article_body = soup.find(class_='article__body')
  if article_body is None:
    article_body = soup.find(class_='article-body')

  # Extract the HTML content from the article body
  html_content = str(article_body)

  # Convert the HTML content into markdown
  markdown_content = html2texthtml2text(html_content, bodywidth=0)
  return markdown_content

For the sake of making the content more legible for ChatGPT, we'll make a function that strips links from the markdown content and replaces them with just the text of the link. We can do so using regular expressions, like so:

import re


def strip_links_from_markdown(markdown_content):
  # Make a RegEx pattern
  link_pattern = re.compile(r'\[(.*?)\]\((.*?)\)')

  # Find all matches in markdown_content
  matches = link_pattern.findall(markdown_content)

  for _match in matches:
    full_match = f'[{_match[0]}]({_match[1]})'

    # Replace the full link with just the text
    markdown_content = markdown_content.replace(full_match, _match[0])

  return markdown_content

Lastly, split the articles by markdown headings into multiple documents.

def split_markdown_by_headings(markdown_content):
  # Define the regular expression pattern to match H1 and H2 headings
  pattern = r'(#{1,2}.*)'

  # Split the Markdown text based on the headings
  sections = re.split(pattern, markdown_content)

  # Combine each heading with its corresponding text
  combined_sections = []
  for i in range(1, len(sections), 2):
    heading = sections[i].strip()  # Get the heading from sections[i]
    text = sections[i + 1].strip() if i + 1 < len(
        sections) else ''  # Get the text from sections[i + 1] if it exists, otherwise use an empty string
    combined_section = f"{heading}\n{text}"  # Combine the heading and text using a newline character
    combined_sections.append(combined_section)  # Add the combined section to the list

  if len(combined_sections) == 0:
    combined_sections = [markdown_content]

  return combined_sections

We now have all of the logic to both scrape and process articles from the Reference. We can put it together in a function, create_documents_from_webpage:

def create_documents_from_webpage(url):
  # Wrap function in a try-except block to handle any potential exceptions
  try:
    soup = scrape_webpage(url)

    # Extract the title from the article
    title = soup.find('h1').get_text().strip()

    # Convert the webpage HTML to markdown
    markdown_content = webpage_to_markdown(soup)

    # Strip links from the markdown content
    markdown_content = strip_links_from_markdown(markdown_content)

    # Split the article up headings
    contents_by_heading = split_markdown_by_headings(markdown_content)
    docs = []
    for content in contents_by_heading:
      docs.append(Document(title=title, url=url, content=content))
    return docs
  except:
    return []

With this function, we can provide any URL from the Reference, and it will generate a list of documents with content from each of the subheadings.

Generate Databank of Sources

Now that we have all of the logic required to scrape and process articles, we can move on to generating the databank of nutritional sources that will be utilized by the chatbot. To do so, we must iterate through each of the articles on of the pages in the Reference, and save all of the documents created.

We'll put together all of our previous work in the generate_databank function:

def generate_databank():
  docs = []

  # Get all of the links in the Reference
  all_links = get_all_links_from_reference()

  # Iterate through every link in the Reference
  for link in all_links:
    # Add all of the documents created from the article
    docs_from_article = create_documents_from_webpage(link)
    docs.extend(docs_from_article)

  return docs

Feel free to add additional debug comments, such as which link is currently being processed, how many links are left, how many documents were added, etc.

After running generate_databank, we now have a databank of nutrition sources!

Keep in mind that this may be a memory-intensive process, and depending on your machine, you may want to consider running this program in a hosted environment like Google Colab. I was able to run this program on my 2023 M2 Pro MacBook Pro, but results may vary by your specific setup.

Conclusion

Through following the steps in this article, we utilized real-world web-scraping techniques to gather and process information from the internet and create a databank of sources; this databank will be used to provide reliable information to our nutrition chatbot.

We used the BeautifulSoup library and the requests library to crawl through the WebMD Health & Diet Medical Reference, and wrote functions to convert the webpage into markdown text, strip links from the markdown content, and split the articles by markdown headings. Finally, we generated the databank of nutritional sources by iterating through all of the articles on pages in the Reference and saving all of the documents created.

In the next article in this series, we'll explore using sentence-transformers and FAISS to create a vector store index to connect the databank we created to an LLM like ChatGPT.

Firebase is one of the most popular Backend-as-a-Service options for a modern tech stack. In addition to offering a NoSQL database solution called Firestore, the Firebase platform provides solutions for authentication, file storage, hosting, and analytics. The Firebase SDK is available for many platforms, including mobile, Unity, Java, C++, and the web.

One of the major shortcomings of Firebase on the web, however, was its sheer size. According to BundlePhobia, a tool used to determine the size of NPM packages, the firebase Web Javascript package weighs in at 235.5kB when minified & g-zipped. This can result in an additional 0.59s of loading time for some users with slower network connections. For comparison, lodash is another notoriously heavy NPM package yet it only weighs 24.5kB when minified & g-zipped: a 10th of the size of Firebase.

This is a known issue with the Firebase Web Javascript SDK, and has turned many developers away from the product. Especially for developers building products for end-users who may not have access to a fast internet connection, loading a package as large as Firebase was simply not an option for them.

Thankfully, the Firebase team has been hard at work recreating the Firebase Web SDK from the group up. On July 27th, 2021, the official Firebase Blog account announced the pre-release of the a new, modular JavaScript SDK that can be “up to 80% smaller!”

Firebase Web v9 will completely change how web developers use Firebase. With the introduction of a fully overhauled, modular, functional programming style and the inclusion of a Firestore “lite” library, web apps powered by Firebase Web v9 will run faster, loader quicker, and dramatically enhance both the user and developer experience.

With all that said, let’s take a look at some of the radical changes introduced in this new modular Firebase Web SDK.

Side-effect-free imports

Previously, the Firebase Javascript SDK incorporated what is known as side-effect imports. In simplest terms, a side-effect occurs when a function modifies state outside of its provided scope. For example, if function a were to modify global variable x, then function a would produce a side-effect. Side-effect imports effect the state, logic, or functionality of your program without calling any methods or referencing any variables that are exported from the package. The mere presence of the package in your program (via an import statement or require call) can affect the functionality of your program.

The old Firebase Web SDK heavily relied on side-effect imports. For each additional Firebase functionality you wanted to include in your app (authentication, Firestore, cloud storage, analytics, etc.), you had to import an additional package like so:

// main firebase app import
import app from "firebase/app";

// SIDE EFFECT PACKAGES
import "firebase/auth";
import "firebase/firestore";
import "firebase/storage";

If you have experience with working with the old Firebase Web SDK, you might have incorporated lazy-loading for importing the Firebase packages. This solution would decrease the initial load size and time of your web app, but users would still be forced to wait for all of these packages to load before the app became fully functional.

Firebase Web v9 changes all of this. The concept of side-effect packages is non-existent in the new Firebase Web SDK, and all of the packages are completely tree-shakable. That means that only the parts of Firebase that are needed by your app will be imported on the client. This drastically reduces the final bundle size of your app and will lead to much faster loading times!

Native Javascript ES modules

In the new Firebase Web SDK, each individual functionality of Firebase that your app requires is imported separately thanks to the introduction of modular packages. Because the new SDK is built into native Javascript ES modules, you can directly import only the features that your program needs: nothing more, nothing less. For example, let’s say you want to initialize your Firebase app and then watch for auth changes:

// imports with ES modules
import { initializeApp } from "firebase/app";
import { getAuth, onAuthStateChanged } from "firebase/auth";

// initialize firebase app
initializeApp(firebaseConfig);

// watch for auth changes
const auth = getAuth();
onAuthStateChanged(auth, (user) => {
 // deal with authentication changes 
});

The introduction of modular packages in turn results in the introduction of a more functional programming style when working with the Firebase Web SDK.

Functional programming style

If you have ever worked with functional programming languages or libraries, you will be familiar with the advantages that functional programming grants you as a developer. Programs that adhere to the functional programming style often have the advantages of being very intuitive and incredibly test-friendly. Although the old Firebase Web SDK was hardly difficult to comprehend, the new Firebase Web SDK is no less intuitive or beginner-friendly.

To demonstrate the functional programming style introduced by the new modular Firebase packages, let’s look at an example of updating a document in Firestore.

import { getStorage, ref, uploadBytes } from "firebase/storage";


// first, get a reference to the storage bucket for our app
const storage = getStorage();

// then, make a reference to the file
const usersCollection = ref(storage, "files/example.png");

// finally, upload the file to the reference
uploadBytes(usersCollection, file);

As you can see, there is a lot of function nesting present in this code example—the result of one function is passed as the argument to another function, whose result is passed to the argument of another function, and so on. This is in stark contrast to the method chaining approach used by the old Firebase Web SDK.

To summarize, the code used with the new Firebase SDK functional languages like F# or Scala (or functional libraries like Ramda and RxJS), whereas the code used with the old Firebase Web SDK resembles that of Java or C++.

Firestore Lite

Firestore is an incredibly powerful and useful database service. It provides a lot of features, many of which aren’t actually utilized in all web apps that use Firestore. Many developers simply use Firestore as an easy-to-implement NoSQL database that handles many of the complexities of operating a database on both the client- and server-side. To that extent, a lot of web apps don’t need the realtime updates capability of Firestore; they just need access to one-time document and collection queries.

The Firebase team recognizes this valid use case, and has addressed it with the introduction of a new library: Firestore Lite. The Firestore Lite library is up to 80% lighter than the old Firestore v8 library. All of the features of Firestore that you have grown to love and take full advantage of, minus realtime updates, are available in the Firestore Lite library. This is a big win for the Firebase Web community because your apps can now be more performant and less bloated with unused code!

Compatibility

The new Firebase Web v9 SDK makes it easy to progressively upgrade from the v8 SDK. The firebase package provides a compat library to make migrating from v8 to v9 easy and incremental. For all of the places in your codebase where you aren't ready to make the full switch to Firebase Web v9, you can take advantage of the compat library and incrementally upgrade parts of your codebase until you no longer need to use the compat library functionality.

The main drawback for this is that you won’t experience all of the bloat and load time reducing features of the new v9 SDK when using the compat library. The compat library still relies on side-effect imports, so you will have to deal with those like you would with the Firebase Web v8 SDK.

Conclusion

If you have ever worked with Firebase on the web before, the future of Firebase should really excite you. The introduction of this new modular Firebase Web v9 SDK changes everything in terms of developing with Firebase on the web. From making your apps less bloated to improving the experiences of both the developer and the end-user, the new Firebase Web v9 modular SDK removes one of the biggest downsides to using Firebase, and will revolutionize the future of Firebase-powered web apps!