How I Generated 100k webpages for my Side Project using AI

Date: 2024-10-11 | create | tech | artificial-intelligence | fullstack-projects | projects |

I'm currently building FullstackProjects - an idea generator for side projects you might want to build. The initial version consisted of mashing brands / nouns together but I wanted to see if we could take this further by using AI.

In this post I'll share how I utilized Open AI's API to generate 100k+ webpages for these ideas.

Generating a Project Webpage with AI

A webpage is really just a display of information. So in order to have a webpage we need info to display on it.

FullstackProjects' ideas are really just phrases and often pretty bad / meaningless nonsense. So we use AI to give better context for what an idea based on this might look like, then we display that on a page.

The prompt I'm using asks the AI to:

  • Give a description of the project idea
  • Provide ratings and explanations for it
  • Return this in a json format my code understands

This makes AI do the thinking part and then give us back those thoughts in a way my code can utilize.

The prompt I'm currently using looks like this:

You are an expert entrepreneur. It's your job to rate project ideas and give details about how you would build them.

The user will give you an idea and you will rate them on a scale of 1-5 for several categories. 1 means this is not a very good idea in this category. 5 means this is a gret idea in this category. The categories are:
* business - The potential of the idea to be turned into a business
* side_project - How easy this would be to build as a side project (one person in one week)
* learning - How much someone would learn building this with a focus on business / technical skills that are useful to solo founders and builders

You will also provide:
* A description - A brief description of what this project might look like. Focus on a Minimum Viable Product that could be built and tested in a few weeks. 
* An explanation of your ratings

Your response should be in the form:

{
    "project_idea": "[The idea you were given]",
    "description": "[Description of project]",
    "ratings": {
        "business": 4,
        "side_project": 3,
        "learning": 5
    },
    "explanations": {
        "business": "[explanation]",
        "side_project": "[explanation]",
        "learning": "[explanation]"
    }
}

I am not an AI expert so there are probably better ways to do this but this is what I've got so far and seems to follow general best practices for LLM prompting.

  • Set perspective
  • Set criteria
  • Instructions
  • Example response

A full overview of how I integrated my system with the AI APIs is outside the scope of this project but I have an overview of the approach including code in this post: How to call Open AI's API with F# (GPT-4o)

Generating 100k webpages with AI

Now that we have an understanding of how we might generate a single webpage, let's look at how we can scale this up to 100k webpages.

A naive approach would be to simply loop over all the ideas and generate descriptions for them. This is basically what we'll do but with a slight modification so we aren't constantly hitting Open AI's rate limits.

The approach I took was to do a throttled loop, generating about 100 webpages a minute. I built this using F#'s mailbox processor (basically an actor / cron task).

On each run it:

  • Looks up 100 ideas in my DB
  • Generates for the ones without webpages
  • Tells itself to run in a minute

Pretty simple and works p well!

FullstackProjectsContentGenerator:

type FullstackProjectsContentGenerator(serviceTree: FullstackProjectsServiceTree, initialSeconds: int, countToGenerate: int) =

    let mailbox = MailboxProcessor<FullstackProjectsContentGeneratorMessage>.Start(
        fun inbox -> 
            let rec loop() = async {
                let! message = inbox.Receive()
                
                match message with 
                | TickSeconds waitTime ->
                    printfn "FullstackProjectsContentGenerator - Received message. Current wait time: %A seconds" waitTime
                    
                    // Wait for the specified time
                    do! Async.Sleep(waitTime * 1000)  // Convert seconds to milliseconds
                    
                    do! 
                        generateContent
                            serviceTree
                            countToGenerate
                        |> Async.AwaitTask

                    // Double the wait time for the next message
                    // let nextWaitTime = waitTime * 2
                    let nextWaitTime = waitTime
                    inbox.Post(TickSeconds nextWaitTime)
                
                // Continue the loop
                return! loop()
            }
            loop()
    )

    // Kick itself off
    do
        mailbox.Post(TickSeconds initialSeconds) |> ignore

How much did this cost me?

Time-wise this took me a few days to put together:

Money-wise:

  • I spent about $26 on this
  • I am using a cheaper model GPT-4o-mini which keeps costs pretty low
  • My guess is this could be cheaper with smarter usage and I'm prob just using it crudely

Next

Overall I had a good time exploring AI for its generative capabilities. I don't think the content itself is very high quality but also the original ideas weren't that good either.

Big picture I still think an internet future with mostly-AI generated content is a bit grim but I do think there are potential positive outcomes from these capabilities. As an example I may share in a future post how I used these ratings to help rank ideas based on their quality.

If you're looking for some random project ideas, checkout FullstackProjects.

If you liked this post you might also like:

Want more like this?

The best / easiest way to support my work is by subscribing for future updates and sharing with your network.