ChatGPT is more than just a prompting and response platform. You can send prompts to ask for help with SEO, but it becomes more powerful the moment that you make your own agent.
I conduct many SEO audits – it’s a necessity for an enterprise site – so I was looking for a way to streamline some of these processes.
How did I do it? By creating a ChatGPT agent that I’m going to share with you so that you can customize and change it to meet your needs.
I’ll keep things as “untechnical” as possible, but just follow the instructions, and everything should work.
I’m going to explain the following steps”
- Configuration of your own ChatGPT.
- Creating your own Cloudflare code to fetch a page’s HTML data.
- Putting your SEO audit agents to work.
At the end, you’ll have a bot that provides you with information, such as:
Custom ChatGPT for SEO (Image from author, May 2025)You’ll also receive a list of actionable steps to take to improve your SEO based on the agent’s findings.
Creating A Cloudflare Pages Worker For Your Agent
Cloudflare Pages workers help your agent gather information from the website you’re trying to parse and view its current state of SEO.
You can use a free account to get started, and you can register by doing the following:
- Going to http://pages.dev/
- Creating an account
I used Google to sign up because it’s easier, but choose the method you’re most comfortable with. You’ll end up on a screen that looks something like this:
Cloudflare Dashboard (Screenshot from Cloudfare, May 2025)Navigate to Add > Workers.
Add a Cloudflare Worker (Screenshot from Cloudfare, May 2025)You can then select a template, import a repository, or start with Hello World! I chose the Hello World option, as it’s the easiest one to use.
Selecting Cloudflare Worker (Screenshot from Cloudfare, May 2025)Go through the next screen and hit “Deploy.” You’ll end up on a screen that says, “Success! Your project is deployed to Region: Earth.”
Don’t click off this page.
Instead, click on “Edit code,” remove all of the existing code, and enter the following code into the editor:
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request));
});
async function handleRequest(request) {
const { searchParams } = new URL(request.url);
const targetUrl = searchParams.get('url');
const userAgentName = searchParams.get('user-agent');
if (!targetUrl) {
return new Response(
JSON.stringify({ error: "Missing 'url' parameter" }),
{ status: 400, headers: { 'Content-Type': 'application/json' } }
);
}
const userAgents = {
googlebot: 'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.6167.184 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
samsung5g: 'Mozilla/5.0 (Linux; Android 13; SM-S901B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Mobile Safari/537.36',
iphone13pmax: 'Mozilla/5.0 (iPhone14,3; U; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/19A346 Safari/602.1',
msedge: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246',
safari: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9',
bingbot: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/',
chrome: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',
};
const userAgent = userAgents[userAgentName] || userAgents.chrome;
const headers = {
'User-Agent': userAgent,
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding': 'gzip',
'Cache-Control': 'no-cache',
'Pragma': 'no-cache',
};
try {
let redirectChain = [];
let currentUrl = targetUrl;
let finalResponse;
// Follow redirects
while (true) {
const response = await fetch(currentUrl, { headers, redirect: 'manual' });
// Add the current URL and status to the redirect chain only if it's not already added
if (!redirectChain.length || redirectChain[redirectChain.length - 1].url !== currentUrl) {
redirectChain.push({ url: currentUrl, status: response.status });
}
// Check if the response is a redirect
if (response.status >= 300 && response.status < 400 && response.headers.get('location')) { const redirectUrl = new URL(response.headers.get('location'), currentUrl).href; currentUrl = redirectUrl; // Follow the redirect } else { // No more redirects; capture the final response finalResponse = response; break; } } if (!finalResponse.ok) { throw new Error(`Request to ${targetUrl} failed with status code: ${finalResponse.status}`); } const html = await finalResponse.text(); // Robots.txt const domain = new URL(targetUrl).origin; const robotsTxtResponse = await fetch(`${domain}/robots.txt`, { headers }); const robotsTxt = robotsTxtResponse.ok ? await robotsTxtResponse.text() : 'robots.txt not found'; const sitemapMatches = robotsTxt.match(/Sitemap:\s*(https?:\/\/[^\s]+)/gi) || []; const sitemaps = sitemapMatches.map(sitemap => sitemap.replace('Sitemap: ', '').trim());
// Metadata
const titleMatch = html.match(/<title[^>]*>\s*(.*?)\s*<\/title>/i);
const title = titleMatch ? titleMatch[1] : 'No Title Found';
const metaDescriptionMatch = html.match(/<meta\s+name=["']description["']\s+content=["'](.*?)["']\s*\/?>/i);
const metaDescription = metaDescriptionMatch ? metaDescriptionMatch[1] : 'No Meta Description Found';
const canonicalMatch = html.match(/<link\s+rel=['"]canonical['"]\s+href=['"](.*?)['"]\s*\/?>/i);
const canonical = canonicalMatch ? canonicalMatch[1] : 'No Canonical Tag Found';
// Open Graph and Twitter Info
const ogTags = {
ogTitle: (html.match(/<meta\s+property="og:title"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[1] || 'No Open Graph Title',
ogDescription: (html.match(/<meta\s+property="og:description"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[1] || 'No Open Graph Description',
ogImage: (html.match(/<meta\s+property="og:image"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[1] || 'No Open Graph Image',
};
const twitterTags = {
twitterTitle: (html.match(/<meta\s+(name|property)="twitter:title"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Title',
twitterDescription: (html.match(/<meta\s+(name|property)="twitter:description"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Description',
twitterImage: (html.match(/<meta\s+(name|property)="twitter:image"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Image',
twitterCard: (html.match(/<meta\s+(name|property)="twitter:card"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Card Type',
twitterCreator: (html.match(/<meta\s+(name|property)="twitter:creator"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Creator',
twitterSite: (html.match(/<meta\s+(name|property)="twitter:site"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Site',
twitterLabel1: (html.match(/<meta\s+(name|property)="twitter:label1"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Label 1',
twitterData1: (html.match(/<meta\s+(name|property)="twitter:data1"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Data 1',
twitterLabel2: (html.match(/<meta\s+(name|property)="twitter:label2"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Label 2',
twitterData2: (html.match(/<meta\s+(name|property)="twitter:data2"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Data 2',
twitterAccountId: (html.match(/<meta\s+(name|property)="twitter:account_id"\s+content="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"\s*\/?>/i) || [])[2] || 'No Twitter Account ID',
};
// Headings
const headings = {
h1: [...html.matchAll(/<h1[^>]*>(.*?)<\/h1>/gis)].map(match => match[1]),
h2: [...html.matchAll(/<h2[^>]*>(.*?)<\/h2>/gis)].map(match => match[1]),
h3: [...html.matchAll(/<h3[^>]*>(.*?)<\/h3>/gis)].map(match => match[1]),
};
// Images
const imageMatches = [...html.matchAll(/<img\s+[^>]*src="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"[^>]*>/gi)];
const images = imageMatches.map(img => img[1]);
const imagesWithoutAlt = imageMatches.filter(img => !/alt=".*?"/i.test(img[0])).length;
// Links
const linkMatches = [...html.matchAll(/<a\s+[^>]*href="https://www.searchenginejournal.com/create-your-own-chatgpt-agent-for-on-page-seo-audits/546016/(.*?)"[^>]*>/gi)];
const links = {
internal: linkMatches.filter(link => link[1].startsWith(domain)).map(link => link[1]),
external: linkMatches.filter(link => !link[1].startsWith(domain) && link[1].startsWith('http')).map(link => link[1]),
};
// Schemas (JSON-LD)
const schemaJSONLDMatches = [...html.matchAll(/<script[^>]*type="application\/ld\+json"[^>]*>(.*?)<\/script>/gis)];
const schemas = schemaJSONLDMatches.map(match => {
try {
return JSON.parse(match[1].trim());
} catch {
return { error: "Invalid JSON-LD", raw: match[1].trim() };
}
});
// Microdata
const microdataMatches = [...html.matchAll(/<[^>]*itemscope[^>]*>/gi)];
const microdata = microdataMatches.map(scope => {
const typeMatch = scope[0].match(/itemtype=["'](.*?)["']/i);
return {
type: typeMatch ? typeMatch[1] : 'Unknown',
raw: scope[0],
};
});
// Response Headers
const responseHeaders = Array.from(finalResponse.headers.entries());
// Construct final JSON output
return new Response(
JSON.stringify({
targetUrl,
redirectChain,
sitemaps,
metadata: { title, metaDescription, canonical },
headings,
schemas,
openGraph: ogTags,
twitterCards: twitterTags,
images: { total: images.length, withoutAlt: imagesWithoutAlt, imageURLs: images },
links,
microdata,
robotsTxt,
responseHeaders,
//rawHTML: html,
}),
{ headers: { 'Content-Type': 'application/json' } }
);
} catch (error) {
return new Response(
JSON.stringify({ error: error.message }),
{ status: 500, headers: { 'Content-Type': 'application/json' } }
);
}
}You have two things to do at this point:
- Copy the URL to your worker.
- Deploy your worker.
This is the URL you’ll need in the next section. You can find it here:
Cloudflare Worker Preview (Screenshot from Cloudfare, May 2025)Be sure to hit “Deploy” before exiting the screen. If you want to see the basic output at this stage, you can.
Paste your URL into your browser and add the following after the /?url=https://www.searchenginejournal.com.
Your URL will look something like this: https://YOURURL.workers.dev/?url=https://searchenginejournal.com.
Swap out the URL to one of your choosing to test this out. It’s not the “prettiest” at this stage, so it’s now time to move on to the fun part of configuring your own GPT.
Note: This worker is not working with JavaScript-rendered sites. But for everyone else using this agent, it should work fine. Feel free to improve it to work with JavaScript rendering.
Configuring Your Own GPT To Mimic My Agent
You need to first configure your GPT, and you can do this by opening ChatGPT and going to “Explore GPTs,” or you can simply follow this link.
Explore GPTs (Screenshot from ChatGPT, May 2025)You’ll then go to “+ Create“:
Create GPT (Screenshot from ChatGPT, May 2025)Now, it will say “Create” and “Configure.” Go to Configure and start plugging in your information.
You can feel free to change things up a bit, but I recommend following everything I’m adding below to build the foundation of your auditor.
You’ll add what I’m going to list in this section:
Configure GPT (Screenshot from ChatGPT, May 2025)Name: OnPage SEO Audit
Description: Analyze SEO performance of any webpage using custom user-agents. Get detailed insights into metadata, redirect chains, Open Graph tags, Twitter Cards, sitemaps, and more. Perfect for SEO professionals and developers.
Instructions:
Trigger: When a user submits a URL (required) and an optional user-agent:
Instruction: Use the provided inputs to make an API request to retrieve SEO data. Default to the chrome user-agent if not provided.
Trigger: When the API returns valid data:
Instruction: Analyze the data and provide:
A summary of the page's SEO performance.
Actionable suggestions for improvement, categorized into metadata, technical SEO, and content.
Follow-up questions to clarify user priorities or goals, such as:
"Do you have specific goals for this page, such as improving search visibility, click-through rates, or user engagement?"
"Would you like me to focus on technical SEO or content-related improvements first?"
Example Response:
"The page's meta description is missing, which can impact click-through rates. Would you like me to suggest a draft description?"
Trigger: When the API returns HTTP 403:
Instruction:
Retry the request using the chrome user-agent.
If the issue persists:
Notify the user of the problem.
Suggest verifying the URL or user-agent compatibility.
Trigger: When the API returns a 400 error:
Instruction:
Clearly explain the error and provide actionable steps to resolve it (e.g., verify the URL format or ensure required parameters are provided).
Trigger: When data is incomplete or missing:
Instruction:
Request additional information from the user or permission to explore fallback data sources.
Example Follow-Up:
"The API response is missing a meta description for this page. Can you confirm if this was intentional, or should we explore other sources?"
Additional Guidelines:
Include:
A categorized summary of the page's SEO performance (e.g., metadata, technical SEO, content).
A prioritized list of recommended actions.
Visual examples or detailed explanations, when applicable.
Proactively address multiple detected issues with follow-up questions:
"The page has several critical issues, including missing Open Graph tags and a non-canonical URL. Would you like me to prioritize recommendations for social media or canonicalization first?"
Conversation starters
User-Agent: Googlebot, URL: https://example.com
Analyze the SEO details for https://example.com using Googlebot.
Analyze the page using the Samsung Galaxy S22 user-agent.
What metadata is available for https://example.com with Chrome?
Capabilities
Web Search
Code Interpreter & Data Analysis
Your configuration at this point should look something like this:
GPT Configurations Populated (Screenshot from ChatGPT, May 2025)Review all of these fields and see if you have populated each properly before moving on to Create new action.
Navigate to Authentication and choose None.
GPT Authentication (Screenshot from ChatGPT, May 2025)Add the following ChatGPT action coding to the Schema field, but be sure to change the servers > url field to your own URL. I’ll name it “https://CHANGETOYOURURL.com/” so that it’s easy for you to find.
{
"openapi": "3.1.0",
"info": {
"title": "Enhanced SEO Analysis and Audit API",
"description": "Fetch SEO data for analysis. Use the returned data to generate actionable SEO recommendations using AI or experts.",
"version": "1.2.0"
},
"servers": [
{
"url": "https://CHANGETOYOURURL.com/",
"description": "Base URL for Enhanced SEO Analysis API"
}
],
"paths": {
"/": {
"get": {
"operationId": "fetchAndAuditSEOData",
"summary": "Fetch and Audit SEO Data",
"description": "Retrieve SEO analysis data using a user-agent and URL and perform a basic SEO audit.",
"parameters": [
{
"name": "user-agent",
"in": "query",
"description": "The user-agent for the request.",
"required": true,
"schema": {
"type": "string",
"enum": ["chrome", "googlebot", "iphone13pmax", "samsung5g"]
}
},
{
"name": "url",
"in": "query",
"description": "The URL of the webpage to analyze.",
"required": true,
"schema": {
"type": "string",
"format": "uri"
}
}
],
"responses": {
"200": {
"description": "Successful response with audit results",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"metadata": {
"type": "object",
"properties": {
"title": { "type": "string" },
"metaDescription": { "type": "string" },
"canonical": { "type": "string" }
}
},
"redirectChain": {
"type": "array",
"items": {
"type": "object",
"properties": {
"url": { "type": "string" },
"status": { "type": "integer" }
}
}
},
"openGraph": {
"type": "object",
"properties": {
"ogTitle": { "type": "string" },
"ogDescription": { "type": "string" },
"ogImage": { "type": "string" }
}
},
"twitterCards": {
"type": "object",
"properties": {
"twitterTitle": { "type": "string" },
"twitterDescription": { "type": "string" },
"twitterImage": { "type": "string" }
}
},
"sitemaps": {
"type": "array",
"items": { "type": "string" }
},
"robotsTxt": {
"type": "string"
},
"audit": {
"type": "object",
"properties": {
"issues": {
"type": "array",
"items": { "type": "string" }
},
"recommendations": {
"type": "array",
"items": { "type": "string" }
}
}
},
"auditSummary": {
"type": "array",
"items": {
"type": "string"
}
},
"nextSteps": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
},
"400": {
"description": "Bad Request",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"error": { "type": "string" }
}
}
}
}
}
}
}
}
}
}
You should see it under “Available actions: fetchAndAuditSEOData.”
Under Privacy Policy, add a link to your own privacy policy.
Finally, tap “Create” at the top right and follow the prompts.
You can now view your GPT; it will be similar to this On-Page SEO Audit GPT.
Testing Your GPT And Getting Familiar With Your Options
You’ve come a long way with your own GPT, and it’s time to test things out.
Tap on the second tile, “Analyze the SEO details.” It will default to example.com, but you can prompt it to test a URL you like.
Let’s try: netflix.com by prompting, “Use netflix.com as the URL.”
Screenshot from ChatGPT, May 2025You can now experiment with any of the GPT options available to see how everything works together.
Customizing Your GPT Further
You may want to customize your GPT further by going back to where you created it and updating a few things.
Update your conversation starters to adjust:
- User-agents.
- Edit instructions to better meet your needs by adding triggers and responses.
If you like, go into the coding for your worker and add to “const userAgents” to add more user agents to the list.
You can now go to your custom GPT agent and simply prompt it on what page to scan next.
It’s easy to do this by prompting something like this: “Change the URL to THEDESIREDURL,” and your agent will get to work for you.
In Summary
This custom GPT agent is just one example of how combining ChatGPT with Cloudflare Workers can streamline core SEO tasks.
Experiment with the Agent and change it for your needs.
AI has the capacity to help with many tasks and it’s here to stay. So, embracing this technology to be an assistant and an effective tool has possibilities to help SEOs be more efficient at scale.
More Resources:
Featured Image: ImageFlow/Shutterstock
