- For individual users
- Instant access to solutions
- Ask your tech questions
- Start your 30-day Free Trial
Main Topics
Browse All TopicsQuestion
WGET Spider Single Page
Hi all, im currently working on scripts that need to be able to crawl a single web page, and list out the page sizes and load times of each section / component. E.g. CSS download times, images, javascript files.
It would need to be using the command line and i believe wget to be the best option for actually crawling all page components.
I can see how to use wget --spider to get the page, + all images, css etc from a page, but then it also crawls all internal site links on that page too, and in turn all the parts of that page, which is NOT what i need as i only want it to do this on a single page basis!
Heres what i have so far, but im sure there must be a wget setting for a single page and its components only.
wget -r -nd --spider -o links.txt -np -p http://www.mysite.co.uk/pa
Cheers for any help.
It would need to be using the command line and i believe wget to be the best option for actually crawling all page components.
I can see how to use wget --spider to get the page, + all images, css etc from a page, but then it also crawls all internal site links on that page too, and in turn all the parts of that page, which is NOT what i need as i only want it to do this on a single page basis!
Heres what i have so far, but im sure there must be a wget setting for a single page and its components only.
wget -r -nd --spider -o links.txt -np -p http://www.mysite.co.uk/pa
Cheers for any help.
This question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Subscribe now for full access to Experts Exchange and get
Instant Access to this Solution
- Plus...
- 30 Day FREE access, no risk, no obligation
- Collaborate with the world's top tech experts
- Unlimited access to our exclusive solution database
- Never be left without tech help again
Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.
Your Complete Technology Resource
900+ topic areas (and counting)
Featured Video Tutorials
More Technology Tutorials
Featured Articles
WARNING: 5 Reasons why you should NEVER fix a computer for free.
It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul.
We love the accolades. We love to be seen as the digital white knight who fixed the server, the computer, the email, and anything else that life depends on.
We love it so much, we sometimes make horrible decisions. Sometimes, we work "FOR FREE."
SCCM OSD Basic Troubleshooting
SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispersed nature of the blog posts that deal with some of these issues. This article will help SCCM OSD newbies with some of these issues.
Integrating SCCM and MDT 2008.
For our deployments, we always install MDT 2008 on our SCCM server and select Configure ConfigMgr Integration from the MDT area on the start menu. This gives us:
- Templated task sequences we can import into SCCM
- Additional TS Variables for use in our task sequences
- Additional options and flexibility around computer backup...
Get full access to the help you need.
30-day free trial. Register in 60 seconds.
360° Features
Go Beyond Just Q and A
Got a question?
We've got the answer.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
Need individual assistance?
Our experts are ready to help.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Want to learn from the best?
Read articles from industry experts.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Working on a long term project?
Store your work and research.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
The Latest Technology News and Tips
EE Tech News Blog
Bringing you the latest in technology
| Congressman Proposes Bill to Ban... | 11/28/12 |
| Death to Internet Anonymity: Google... | 11/27/12 |
| TEDx San Luis Obispo | 11/20/12 |
| SharePoint 2010 No-Code Solutions... | 11/19/12 |
| Keeping Your Passwords And Your Win... | 11/15/12 |
| Fun and Games with Hackers: How to... | 11/15/12 |
EE Tech News Podcasts
Listen while you work
New Solutions Everyday
Because of our active community of experts, on average new questions receive their first comment in under 30 minutes.
Top Linux Questions
- VNC Issue with OpenSuse …379 Views
- Need help to configures multi…354 Views
- Is Ubuntu 12.10 support…351 Views
- Problem with Nagios plugin in…319 Views
- UBUNTU 12.10 CRASHI…249 Views
- sftp transfer237 Views
- Fedora Linux: Detecting US…216 Views
- wordpress internal server er…211 Views
- Installing a test SSL certificat…201 Views
- Websphere : xml output not…196 Views
- How to test LDAP connectio…1,995 Views
- Why does my keyboard not …1,602 Views
- Linux Error error while loadin…1,431 Views
- Install and configure vnc ser…1,116 Views
- RHEL 6 and /etc/security…1,069 Views
- mount -o rw,remount ...1,044 Views
- Apache Virtual host configur…1,016 Views
- SP2-0750: You may n…770 Views
- keyboard emulator768 Views
- How to start ftp service gss…763 Views
Stay ahead of the curve with Experts Exchange
30-day free trial. Register in 60 seconds.
Hear what our users are saying
"The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange."
"Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google."
"Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem."
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the Community
Give a little. Get a lot.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
3 Ways to Join
Business Accounts
- Perfect for organizations
- Multiple users
- Save over 36%
- Create a Business Account
Answer for Membership
- Earn free access
- Showcase your knowledge
- No credit card required
- Sign Up to Answer