I am trying to scrape some data from this site https://www.itjobswatch.co.uk/ and whilst I can automate extraction at a high level for multiple pages (eg. all permanent vacancies advertised in the past 6 months) using a function+list for page numbers, what I cannot do is create a function with multiple parameters so that I can extract (with a single query) data for permanent jobs for the past 6 months and past 3 months (there is a filter button on the site).
... or even better
all vacs perm in last 3 months
+
all vacs contract in last 3 months
+
all vacs perm in last 6 months
+
all vacs contract in last 6 months
... with one query/function
Ideally I would then like to use this multi-parameter approach for other selections but at the moment just looking for help please as I cannot find anything addressing this on the web!
Thanks
Pete
Hi Peter,
Have you see this tutorial on scraping data from multiple web pages? I can see from the URL structure that you have parameters in the string that you can specify the category and period. You'd have to run separate queries for each period because the website doesn't allow you to specify both 6 and 3 month periods. Likewise for permanent and contracts. i.e. you can only scrape what the website can display on a page. However, you can then append the queries together to make one table.
Mynda
