Proxies for collecting AI training data

Building a dataset from the public web means crawls that last for weeks and cover several languages. Run them on ProxyWorld unlimited residential and you can set the cost for the whole period on day one, sending requests from 60 countries.

Per GB pricing makes budgets hard

With thousands of domains, page size is not up to you. Mix in a few sites with heavy photos and scripts and your GB use can grow several times past the plan.

Unlimited is priced only by period and connection tier. So you can write down a 90 day crawl budget as one amount before the first request goes out.

Collecting each language separately

Many sites check which country a visitor's IP is in and show the page in that country's language. Put a country code such as -cc-es (Spain) or -cc-id (Indonesia) in the username and requests leave from an IP in that country.

Keep a separate job queue per language and give each a different username. The crawler code needs no changes, and results pile up separately by language.

Working out a 90 day crawl

The 200 connection, 100 Mbps tier on Residence (90 days) is $3.31 a day, $297.90 in total. Buying Layover (1 day) 90 times for the same stretch would be $391.50, so Residence costs $93.60 less.

For a bigger job, buy the 1,000 connection, 200 Mbps tier on Residence. It is $7.19 a day, $647.10 in total. In the dashboard you can enter any number of days from 1~365, so you can buy exactly what your crawl schedule needs.

Rules to follow while collecting

Collect public content only, and follow robots.txt and each site's terms of use. Check the copyright and privacy rules that apply to AI training, and delete personal data you do not need before you store anything.

Even when the proxy has room to spare, set a request cap per domain. You will be going back to the same sites for the next version of the dataset.

How to set up ProxyWorld for AI training data

1
Top up in crypto for the amount the crawl period needs. One top-up can be up to $5,000.
2
Buy unlimited residential for the number of days you will crawl. Residence (90 days) has the lowest daily rate.
3
Create one username per language, such as pwexample1-zone-unl-cc-de, and connect each to a job queue.
4
Connect the crawler to gate.proxyworld.shop:6100 and keep concurrent connections within the tier you bought.

Frequent questions

Can one unlimited plan cover several countries?

Yes. One plan works in all 60 countries. You choose the country for each request in the username.

Do crawls need sticky sessions too?

Usually not. Changing the IP per request is what spreads the load evenly. Use -sid only on sites where cookies must carry across several pages.

Which coins can I top up with?

We accept USDT, USDC, Bitcoin, Ethereum, BNB, Solana, TRON and Litecoin. The amount goes into your USD balance.

The crawl is over and I have balance left.

Balance stays with no expiry. Use it for the next crawl or for another product.