David M.

Yeah! That's incredibly helpful. How in the world did you compile this? It's the most comprehensive understanding of LLM crawling I've seen. Super grateful you shared. Thank you!

Cloudfare and Squarespace both allow options that are similar to robots.txt to allow AI crawling. The JS stuff you mentioned though, I had no idea. This is great to understand how LLMs find companies. Seems like the next frontier of finding qualified folks, but how relevant will they be?

Github is available at: https://github.com/DurangoDavid/SOW

I'd like to take a look. If you'd like, I've built an SOW/Project builder that will create detailed plan if you input a detailed problem statement and stack. It's free and in beta (and you can run it locally on your own machine), but you can give it a shot a https://gtmharmony.io/sow